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Chapter 1 

Introduction 

1 .1 Overview of the TIMSS 2011 User Guide and International Database 

TIMSS measures trends in mathematics and science achievement at the fourth and eighth grades in 
participating countries around the world, while also monitoring curricular implementation and 
identifying promising instructional practices. 1 Conducted on a regular 4-year cycle, TIMSS has 
assessed mathematics and science in 1995, 1999, 2003, 2007, and 2011. TIMSS collects a rich array 
of background information to provide comparative perspectives on trends in mathematics and 
science achievement in the context of different educational systems, school organizational 
approaches, and instructional practices. 

To support and promote secondary analyses aimed at improving mathematics and science 
education at the fourth and eighth grades, the TIMSS 2011 International Database makes available to 
researchers, analysts, and other users the data collected and processed by the TIMSS project. This 
database comprises student achievement data as well as student, home, teacher, school, and 
curricular background data for 63 countries and 14 benchmarking participants. 2 Across both grades, 
the database includes data from 608,641 students, 49,429 teachers, 19,612 school principals, and the 
National Research Coordinators of each country. All participating countries gave the IEA permission 
to release their national data. 

For countries that participated in previous assessments, TIMSS 2011 (the fifth data collection in 
the TIMSS cycle of studies) provides trends for up to five cycles at the eighth grade — 1995, 1999, 
2003, 2007, and 2011 — and data over four points in time at the fourth grade — 1995, 2003, 2007, and 
2011. In countries new to the study, the 2011 results can help policy makers and practitioners assess 
their comparative standing and gauge the rigor and effectiveness of their mathematics and science 
programs. Details of the assessments conducted in 201 1 can be found in the TIMSS 2011 
International Reports (Mullis, Martin, Foy, & Arora, 2012; and Martin, Mullis, Foy, & Stanco, 
2012 ). 

TIMSS 2011 was an ambitious and demanding study, involving complex procedures for drawing 
student samples, assessing students’ achievement, analyzing the data, and reporting the results. In 
order to work effectively with the TIMSS data, it is necessary to have an understanding of the 
characteristics of the study, which are described fully in Methods and Procedures in TIMSS and 
PIRLS 2011 (Martin & Mullis, 2012). It is intended, therefore, that this User Guide be used in 


The Trends in International Mathematics and Science Study is a project of the International Association for the Evaluation of Educational 
Achievement (IEA); TIMSS together with PIRLS, the Progress in International Reading Literacy Study, comprise IEA’s core cycle of studies. 

2 The TIMSS 201 1 International Database also includes data from the PIRLS 2011 “Learning to Read” survey (a home questionnaire administered to 
parents) for countries that administered the TIMSS and PIRLS 2011 fourth grade assessments to the same sample of students. 
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conjunction with the Methods and Procedures documentation. Whereas the User Guide describes the 
organization and content of the database, the Methods and Procedures documentation provides the 
rationale for the techniques used and for the variables created in the process of data collection and 
compilation. 

1 .2 Overview of the TIMSS 2011 User Guide and International Database 

This User Guide describes the content and format of the data in the TIMSS 2011 International 
Database. In addition to this introduction, the User Guide includes the following three chapters: 

• Chapter 2 — This chapter introduces the IEA International Database (IDB) Analyzer software 
(IEA, 2012) and presents examples of analyses with the TIMSS 2011 data using this software in 
conjunction with SPSS (IBM Corporation, 2012). 

• Chapter 3 — This chapter presents examples of analyses with the PIRLS 2011 data using the SAS 
statistical software system (SAS Institute, 2011) and the SAS programs and macros included with 
the database. 

• Chapter 4 — This chapter describes the structure and content of the database. 

The User Guide is accompanied by the following four supplements: 

• Supplement 1 — This supplement comprises the international version of all TIMSS 2011 
background questionnaires. 

• Supplement 2 — This supplement describes any adaptations to the questions in the background 
questionnaires made by individual TIMSS participants. 

• Supplement 3 — This supplement describes how derived variables were constructed for reporting 
the TIMSS data. 

• Supplement 4 — This supplement describes the sampling stratification variables for each country. 

The User Guide and its four supplements are available on the TIMSS 2011 International Database 
and User Guide webpage: http://timssandpirls.bc.edu/timss201 1 /international-database. html. 

1 .3 Contents of the TIMSS 2011 International Database 

The entire TIMSS 2011 International Database is available on the TIMSS 2011 International 
Database and User Guide webpage: http://timssandpirls.bc.edu/timss201 1 /international- 
database. html. The Database contains the TIMSS 2011 student achievement data files and student, 
home, teacher, and school background questionnaire data files, along with support materials. 

Exhibit 1 . 1 displays the general structure of the International Database and a brief description of the 
support materials available for download on the International Database and User Guide webpage. 

The TIMSS 2011 International Database also is available for download at the IEA Study Data 
Repository website: http://rms.iea-dpc.org/. The repository allows users to download subsets of files 
and the corresponding support material through customizable queries from all recent IEA studies, 
including TIMSS 2011. 
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Exhibit 1.1: Contents of the TIMSS 2011 International Database 


User Guide 

This User Guide with its four supplements 

Items 

The TIMSS 201 1 released items, item information files, and IRT item parameters 

International Database 

SPSS Data 

TIMSS 201 1 student, home, teacher, and school data files in SPSS format 


SAS Data 

TIMSS 201 1 student, home, teacher, and school data files in SAS format 


Curriculum Data 

TIMSS 201 1 curriculum questionnaires data files 


Codebooks 

Codebook files describing all variables in the TIMSS 201 1 1nternational Database 


Almanacs 

Data almanacs with summary statistics for all TIMSS 201 1 items and background variables 


TCMA 

National item selection data for the Text-Curriculum Matching Analysis (see Appendix C of the 
TIMSS 201 1 International Reports for more details) 


Programs 

SAS and SPSS programs and macros 
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Chapter 2 

Using the IEA IDE Analyzer to Analyze the 
TIMSS 201 1 International Database 


2.1 Overview 

This chapter describes the use of the IEA International Database (IDB) Analyzer software (IEA, 
2012) for analyzing the TIMSS 201 1 data. Used in conjunction with SPSS (IBM Corporation, 2012), 
the IEA IDB Analyzer provides a user-friendly interface to easily merge the various data file types of 
the TIMSS 2011 database and seamlessly takes into account the sampling information and the 
multiple imputed achievement scores to produce accurate statistical results. 

Example analyses will illustrate the capabilities of the IEA IDB Analyzer (version 3.0) to 
compute a variety of statistics, including percentages of students in specified subgroups, mean 
student achievement in those subgroups, correlations, regression coefficients, percentages of students 
reaching benchmark levels, and percentiles of achievement distributions. The examples use student, 
teacher, and school background data to replicate some of the TIMSS 2011 results included in the 
TIMSS 2011 International Reports (Mullis, Martin, Foy, & Arora, 2012; and Martin, Mullis, Foy, & 
Stanco, 2012). 

Users should be able to perform statistical analyses with the IEA IDB Analyzer with a basic 
knowledge of the TIMSS 2011 International Database. Chapter 4 gives a more detailed description 
of the data files contained in the International Database, including their structure and contents, along 
with a description of all the supporting documentation provided with the International Database. 

2.2 The IEA IDB Analyzer 

Developed by the IEA Data Processing and Research Center (IEA DPC), the IEA IDB Analyzer is a 
plug-in for SPSS, a well-known statistical analysis program. The IEA IDB Analyzer enables users to 
combine SPSS data files from IEA’s large-scale assessments and conduct analyses using SPSS 
without actually writing programming code. The IEA IDB Analyzer generates SPSS syntax that 
takes into account information from the sample design in the computation of statistics and their 
standard errors. In addition, the generated SPSS syntax makes appropriate use of plausible values for 
calculating estimates of achievement scores and their standard errors, combining both sampling 
variance and imputation variance. 

The IEA IDB Analyzer consists of two modules — the merge module and the analysis module — 
which are integrated and executed in one common application. The merge module is used to create 
analysis datasets by combining data files of different types and from different countries, and 
selecting subsets of variables for analysis. The analysis module provides procedures for computing 
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various statistics and their standard errors. The latest version (3.0) of the IEA IDB analyzer is 
available for download at the IEA website: http://www.iea.nl/data.html. Once installed, the IEA IDB 
Analyzer can be accessed by using the START menu in Windows: 

Start ■=> All Programs A> IEA ■=> IDB AnalyzerV3 ■=> IEA IDB Analyzer 


2.3 Merging Files with the IEA IDB Analyzer 

The IEA IDB Analyzer uses the SPSS data files available on the TIMSS 2011 International Database 
and User Guide webpage: http://timssandpirls.bc.edu/timss201 1 /international-database. html} The 
TIMSS 2011 data files are disseminated separately by file type and for each country. In addition to 
allowing users to combine like datasets from more than one country for cross-country analyses, the 
merge module allows for the combination of data from different sources (e.g., student, home, 
teacher, and school) into one SPSS dataset for subsequent analyses. Before conducting any statistical 
analyses with the TIMSS 2011 International Database, users should download and copy the contents 
of the International Database either on their computer or on a server. For the purposes of this chapter, 
we will assume all files have been copied to the folder titled “C:\TIMSS201 1\.” 

The following steps will create an SPSS data file with data from multiple countries and/or 
multiple file types: 

1 . Start the IEA IDB Analyzer from the START menu and click the Merge Module button. 

2. Under the Select Data Files and Participants tab in the Select Directory field, browse to the 
folder where all SPSS data files are located. For example, in Exhibit 2.1, all SPSS data files are 
located in the folder titled “C:\TIMSS201 l\Data\SPSS_Data.” The program will automatically 
recognize and complete the Select Study, Select Year, and Select Grade fields and list all 
countries available in this folder as possible candidates for merging. If the folder contains data 
from more than one IEA study, or from more than one grade, the IEA IDB Analyzer will prompt 
users to select files from the desired combination of study and grade for analyses. In Exhibit 2.1, 
the TIMSS 2011 eighth grade is selected. 

3. Click a country of interest from the Available Participants list and click the right arrow button 
( ► ) to move it to the Selected Participants panel. Individual countries can be moved directly to 
the Selected Participants panel by double-clicking on them. To select multiple countries, hold 
the CTRL key of the keyboard when clicking countries. Click the tab-right arrow button ( ►( ) to 
move all countries to the Selected Participants panel. In Exhibit 2.1, Armenia, Australia, 
Bahrain, and Chile are selected. 


The TIMSS 201 1 International Database also is available for download on the IEA Study Data Repository website: http://rms.iea-dpc.org/ 
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Exhibit 2.1 : IEA IDB Analyzer Merge Module — Select Data Files and Participants 


1 IEA IDB Analyzer Merge Module - (Version 3.0.47) 


Select Data Files and Participants | Select File Typ es and Variables 



4. Click the Next > button to proceed to the next step in the Select File Types and Variables tab. 
The software will open the second tab of the merge module, as shown in Exhibit 2.2, to select the 
file types and the variables to be included in the merged data file. 

5. Select the files for merging by checking the appropriate boxes to the left of the window. For 
example, in Exhibit 2.2, the student background data files are selected. 

6. Select the variables of interest from the Available Variables list in the left panel. Note that 
Supplement 1 provides the variable names for all questions in the background questionnaires. 
Variables are selected by clicking on them and then clicking the right arrow ( ► ) button. 
Clicking the tab-right arrow ( ►) ) button selects all variables. Note that there are two tabs: 
“Background Variables and Scores” and “ID and Sampling Variables.” All achievement scores 
and all identification and sampling variables are selected automatically by the IEA IDB 
Analyzer. 
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Exhibit 2.2: IEA IDB Analyzer Merge Module — Select File Types and Variables 


p IEA IDB Analyzer: Merge Module 

(Version 3.0.47) 

1 oiB 

Select Data Files and Participants 

Select File Types and Variables j 





0 Select File Types... 


0 Select Variables... 


Available Variables: 


Selected Variables: 


□ School Background 
O Student Achievement 
[y] Student Background 

O Math Teacher Background 

□ Science Teacher Background 


Background Variables and Scores (294) ID and Sampling Variables (0) } 




Name 

Description 

T 


► BSBG01 

GENNSEX OF STUDENT 

BSBG02A 

GENXDATE OF BIRTH\MONTH 
GENXDATE OF BIRTHXYEAR 

4$BSBG02B 

BSBG03 

GENXOFTEN SPEAK < LANG OF TEST> AT HOME 

ESBGC4 

GENNAMOUNT OF BOOKS IN YOUR HOME 

4$BSBG05A 

GENXHOME POSSESSXCOMPUTER 

4£BSBG05B 

GENXHOME POSSESSXSTUDY DESK 

40BSBGO5C 

GENXHOME POSSESSXBOOKS 

BSBG05D 

GENXHOME POSSESSXOWN ROOM 

4$BSBG05E 

GENXHOME POSSESSMNTERNET CONNECTION 

4$BSBG0SF 

GENXHOME POSSESS\<COUNTRY SPECIFIC 

4$BSBG05G 

GENXHOME POSSESS\<COUNTRY SPECIFIC > 

<2Jbsbgosh 

GENXHOME POSSESS\<COUNTRY SPECIFIC > 

<2$BSBG05! 

GENXHOME POSSESS\< COUNTRY SPECIRC> 

4$BSBG05J 

GENXHOME POSSESS\<COUNTRY SPECIHC> 

4$BSBG05K 

GENXHOME POSSESS\<COUNTRY SPECIFIC 

BSBG06A 

GENXHIGHEST LVL OF EDU\<STMO OR FE GUA... 

^$BSBG06B 

GENXHIGHEST LVL OF EDUVSTFA OR MA GUA... 

4$BSBG07 

GENXHOW FAR IN EDU DO YOU EXPECT TO GO 

BSBG08A 

GEN\<STMO OR FEM GUARD> BORN <COUNT... 

BSBGT3B 

GEN\<STFA OR MA GUARD> BORN IN <COUN... 

4 $ BSBG09A 

GENXBORN IN <COUNTRY> 

4$BSBG09B 

GENXBORN IN <COUNTRY>\AGE OF COMING 




Background Variables and Scores (16) ID and Sampling Variables (23) 


0 Output Files: C:\71MSS2011\Data\BSGALLM5* 


0 | Start SPSS | 






Description 


T 



► XBSMMAT01-05 

1ST TO 5TH PLAUSIBLE VALUE MATHEMATICS* 


XBSSSG01-05 

1ST TO 5TH PLAUSIBLE VALUE SCIENCE’ 


XBSMALG01-05 

1ST TO 5TH PV ALGEBRA- 


XBSMDAT01-05 

IST TO 5TH PV DATA AND CHANCE* 


XBSMNUM01-05 

1ST TO 5TH PV NUMBER* 


X BSMGEO01-05 

1ST TO 5TH PV GEOMETRY* 


XBSSCHE01-05 

1ST TO 5TH PV CHEMISTRY* 


X BSSEAR01-05 

1ST TO 5TH PV EARTH SCIENCE’ 


XBSSBIO01-05 

1ST TO 5TH PV BIOLOGY* 


X BSSPHY01-05 

1ST TO 5THPV PHYSICS’ 


JLBSMKN001-05 

1ST TO 5TH PV MATH KNOWING* 


X BSMAPP01-05 

1ST TO 5TH PV MATH APPLYING* 


X BSMREA01-05 

1ST TO 5TH PV MATH REASONING’ 


X BSSKNO01-05 

1ST TO 5TH PV SCIENCE KNOWING’ 


X BSSAPP01-05 

1ST TO 5TH PV SCIENCE APPLYING* 


XBSSREA01-05 

1ST TO 5TH PV SCIENCE REASONING* 





Return to Main Menu || Help 


TIMSS 2011 Grade 8 


Student Background 


7. Specify the desired name for the merged data file and the folder where it will be stored in the 
Output Files field by clicking the Define/Modify button. The IEA IDB Analyzer also will create 
an SPSS syntax file (*.SPS) of the same name and in the same folder with the code necessary to 
perform the merge. In the example shown in Exhibit 2.2, the merged file BSGALLM5.SAV and 
the syntax file BSGALLM5.SPS both will be created and stored in the folder titled 
“C:\TIMSS201 lYData.” The merged data file will contain all the variables listed in the Selected 
Variables panel to the right. 

8. Click the Start SPSS button to create the SPSS syntax file and open it in an SPSS syntax 
window ready for execution. The syntax file can be executed by opening the Run menu of SPSS 
and selecting the All menu option. The IEA IDB Analyzer will display a warning if it is about to 
overwrite an existing file in the specified folder. 

Once SPSS has completed its execution, it is important to check the SPSS output window for 
possible warnings. If warnings appear, they should be examined carefully because they might 
indicate that the merge process was not performed properly and that the resulting merged data file 
might not be as expected. 
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Merging Student and Teacher Data Files 

The teachers in the TIMSS 2011 International Database do not constitute representative samples of 
teachers in the participating countries. Rather, they are the teachers of nationally representative 
samples of students. Therefore, analyses with teacher data should be made with students as the units 
of analysis and reported in terms of students who are taught by teachers with a particular attribute. 

Teacher data are analyzed by linking the students to their teachers. The student-teacher linkage 
data files (AST/BST) are used for this purpose and the IEA IDB Analyzer will make use of them 
automatically. Thus, to analyze teacher data, it is sufficient to select the Teacher Background file 
types in the Select File Types and Variables tab of the IEA IDB Analyzer merge module. To 
analyze student and teacher background data simultaneously, however, both the Student 
Background and Teacher Background file types must be selected in the Select File Types and 
Variables tab of the IEA IDB Analyzer merge module. The variables of interest need to be selected 
separately for both file types, as follows: 

1 . Click the Teacher Background file type (Math or Science at the eighth grade) so that it appears 
checked and highlighted. The Background Variables and Scores listed in the left panel will 
include all available variables from the teacher background data files. 

2. Select the variables of interest from the left panel and click the right arrow ( ► ) button to move 

these variables to the Selected Variables panel on the right. Click the tab-right arrow ( ) 

button to select all available variables. 

3. If necessary, click the Student Background file type and selecting the variables of interest from 
the Background Variables and Scores panel on the left in the same manner. 

4. Specify the folder and merged data file name in the Output Files field, as described earlier. 

5. Click the Start SPSS button to create the SPSS syntax file that will produce the required merged 
data file, which can then be run by opening the Run menu of SPSS and selecting the All menu 
option. 

Merging Student and School Data Files 

Because TIMSS 2011 has representative samples of schools, it is possible to compute reasonable 
statistics with schools as units of analysis. However, the school samples were designed to optimize 
the student samples and the student-level estimates. For this reason, it is preferable to analyze 
school-level variables as attributes of the students, rather than as elements in their own right. 
Therefore, analyzing school data should be done by linking the students to their schools. 

To merge the student and school background data files, select both the Student Background and 
School Background file types in the Select File Types and Variables tab of the IEA IDB Analyzer 
merge module. The variables of interest to be included in the merged data file are selected separately 
by file type, as described above in “Merging Student and Teacher Data Files” and using the same set 
of instructions. 
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Merged Data Files for the Examples 

To conduct the analysis examples presented in this chapter, the following merged data files should be 
created with all available background variables and achievement scores selected: 

BSGALLM5.SAV — Merged eighth grade student background data files for all countries. 

BTMALLM5.SAV — Merged eighth grade mathematics teacher background data files for all 
countries. 

BCGALLM5.SAV — Merged eighth grade school and student background data files for all countries. 

2.4 Performing Analyses with the IEA IDB Analyzer 

The analysis module of the IEA IDB Analyzer can perform statistical analyses on any files created 
using the merge module. The following statistical procedures are available in the analysis module of 
the IEA IDB Analyzer. 

Percentages and Means 

Compute percentages, means, and standard deviations for selected variables by subgroups defined by 
grouping variable(s). 

Percentages Only 

Compute percentages by subgroups defined by grouping variable(s). 

Regression 

Compute regression coefficients for selected independent variables to predict a dependent variable 
by subgroups defined by grouping variable(s). Plausible values can be included as dependent or 
independent variables. 

Correlations 

Compute means, standard deviations, and correlation coefficients for selected variables by subgroups 
defined by grouping variable(s). 

Benchmarks 

Compute percentages of students meeting a set of user-specified achievement benchmarks, in 
particular the TIMSS International Benchmarks, by subgroups defined by grouping variable(s). 

Percentiles 

Compute the score points that separate a given proportion of the distribution of achievement scores, 
by subgroups defined by the grouping variable(s). 

All statistical procedures offered in the analysis module of the IEA IDB Analyzer make 
appropriate use of sampling weights, and standard errors are computed using the jackknife repeated 
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2 

replication (JRR) method. Percentages, means, regressions, and correlations may be specified with 
or without achievement scores. When achievement scores are used, the analyses are performed five 
times (once for each plausible value) and the results are aggregated to produce accurate estimates of 
achievement and standard errors that incorporate both sampling and imputation errors. To conduct 
analyses using achievement scores, select the Use PVs option from the Plausible Value Option 
drop-down menu. The various variables required to perform an analysis are input into specific 
variable fields according to their purpose. 

Grouping Variables 

This is a list of variables to define subgroups. The list must consist of at least one grouping variable. 
By default, the IEA IDB Analyzer includes the variable IDCNTRY used to distinguish the 
participating countries. Additional variables may be selected from the available list. If the 
Exclude Missing from Analysis option is checked, only cases that have non-missing values in the 
grouping variables will be used in the analysis. If it is not checked, missing values become reporting 
categories. 

Analysis Variables 

This is a list of variables for which means or percentages are to be computed. More than one analysis 
variable can be selected. To compute means for achievement scores, it is necessary to select the 
Use PVs option in the Plausible Value Option drop-down menu, and select the achievement scores 
of interest in the Plausible Values field. 

Plausible Values 

This section is used to identify the set of plausible values to be used when achievement scores are the 
analysis variable for computing percentages and means. Select the Use PVs option in the Plausible 
Value Option drop-down menu before specifying the achievement scores in the Plausible Values 
field. 

Independent Variables 

This is a list of variables to be treated as independent variables for a regression analysis. More than 
one independent variable can be selected. Achievement scores also can be included as an 
independent variable. To specify achievement scores as the independent variable, it is necessary to 
select the Use PVs option in the Plausible Value Option drop-down menu and select the 
achievement scores of interest in the Plausible Values field. 

Dependent Variable 

This is the variable to be used as the dependent variable when a regression analysis is specified. Only 
one dependent variable can be listed and can be either a background variable or the set of five scores 
associated with an achievement scale. To use achievement scores as the dependent variable, it is 


2 See Methods and Procedures in TIMSS and PIRLS 2011 (Martin & Mullis, 2012). 
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necessary to select the Use PVs option in the Plausible Value Option drop-down menu and select 
the achievement scores of interest in the Plausible Values field. 

Weight Variable 

This is the sampling weight variable that will be used in the analysis. The IEA IDB Analyzer 
automatically selects the appropriate weight variable for analysis based on the file types included in 
the merged data file. Generally, this will be TOTWGT, but SENWGT and HOUWGT also are 
available for student-level analyses with student or school data. MATWGT will be used when 
analyzing mathematics teacher data, SCIWGT when analyzing science teacher data. Chapter 4 
provides more information on the TIMSS sampling weights. 

Achievement Benchmarks 

These are the values that will be used as cut points on an achievement scale, selected in the Plausible 
Values section, for computing the percentages of students meeting the specified benchmarks. 
Multiple cut points can be specified, each separated by a blank space. It is necessary to select the Use 
PVs option in the Plausible Value Option drop-down menu and select the achievement scores of 
interest in the Plausible Values field. 

Percentiles 

These are the percentiles that will be calculated from the distribution of the achievement scores 
selected in the Plausible Values section. Multiple percentiles can be specified, each separated by a 
blank space. It is necessary to select the Use PVs option in the Plausible Value Option drop-down 
menu and select the achievement scores of interest in the Plausible Values field. 

2.5 TIMSS Analyses with Student-level Data 

Many analyses of the TIMSS 2011 International Database can be undertaken using only student- 
level data. This section presents examples of actual analyses used to produce exhibits from TIMSS 
2011 International Results in Mathematics . Examples of regression analyses and computing 
percentages of students reaching the TIMSS International Benchmarks also are included in this 
section. 

A first example computes national average achievement, whereas a second example computes 
national average achievement by gender. In both cases, the IEA IDB Analyzer uses the sampling 
weights, implements the jackknife repeated replication method to compute appropriate sampling 
errors, effectively performs the computations five times (once for each plausible value), and 
aggregates the results to produce accurate estimates of average achievement and standard errors that 
incorporate both sampling and imputation errors. A third example expands on the second example by 
perfonning a test of significance on the gender difference using regression. A fourth example 
computes the percentages of students reaching the TIMSS International Benchmarks. Finally, a fifth 
example computes the average scale score for one of the newly developed contextual scales, along 
with the percentages of students, with their average achievement, for the categories of the scale’s 
corresponding index. 
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Student-level Analysis with Achievement 

In our first example, we want to replicate the analysis of the overall distribution of mathematics 
achievement. These results are presented in Exhibit 1.2 of TIMSS 2011 International Results in 
Mathematics and are repeated here in Exhibit 2.3. Because the results in this exhibit are based on 
plausible values, we must ensure that we include them when we create the file using the merge 
module, and also indicate that our analysis will make use of achievement scores. 

The Percentages and Means statistic type with the Use PVs option selected will compute 
percentages and average mathematics achievement based on plausible values and their respective 
standard errors. After creating the merged data file BSGALLM5, the analysis module of the IEA 
IDB Analyzer is used to perform the analysis using the following steps: 

1 . Open the Analysis Module of the IEA IDB Analyzer. 

2. Select the merged data file BSGALLM5 as the Analysis File by clicking the Select button. 

3. Select TIMSS (Using Student Weights) as the Analysis Type. 

4. Select Percentages and Means as the Statistic Type. 

5. Select Use PVs as the Plausible Value Option. 

6. The variable IDCNTRY is selected automatically as Grouping Variables. No additional 
grouping variables are needed for this analysis. 

7. Specify the achievement scores to be used for the analysis and activate them by clicking the 
Plausible Values field. Select BSMMAT01-05 from the list of available variables and move it to 
the Plausible Values field by clicking the right arrow ( ► ) button in this section. 

8. The Weight Variable is selected automatically by the software; TOTWGT is selected by default 
because this example analysis uses student background data. 

9. Specify the name and the folder of the output files in the Output Files field by clicking the 
Define/Modify button. 

10. Click the Start SPSS button to create the SPSS syntax file and open it in an SPSS syntax 
window. The syntax file can be executed by opening the Run menu of SPSS and selecting the 
All menu option. If necessary, the IEA IDB Analyzer will display a prompt to confirm the 
overwriting of existing files. 
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Exhibit 2.3: Exhibit of Example Student-level Analysis with Achievement, Taken from 

TIMSS 201 7 International Results in Mathematics (Exhibit 1.2) 


Exhibit 1 .2: Distribution of Mathematics Achievement 


TIMSS 2011 
Mathematics 


Country 

Average 
Scale Score 

Korea, Rep. of 

613 (2.9) 

O 

2 Singapore 

611 (3.8) 

O 

Chinese Taipei 

609 (3.2) 

O 

Hong Kong SAR 

586 (3.8) 

O 

Japan 

570 (2.6) 

O 

2 Russian Federation 

539 (3.6) 

O 

3 Israel 

516 (4.1) 

O 

Finland 

514 (2.5) 

O 

2 United States 

509 (2.6) 

O 

♦ England 

507 (5.5) 


Hungary 

505 (3.5) 


Australia 

505 (5.1) 


Slovenia 

505 (2.2) 

O 

1 Lithuania 

502 (2.5) 

TIMSS Scale Centerpoint 

500 

Italy 

498 (2.4) 

New Zealand 

488 (5.5) 

® 

Kazakhstan 

487 (4.0) 

® 

Sweden 

484 (1.9) 

® 

Ukraine 

479 (3.9) 

® 

Norway 

475 (2.4) 

® 

Armenia 

467 (2.7) 

® 

Romania 

458 (4.0) 

® 

United Arab Emirates 

456 (2.1) 

® 

Turkey 

452 (3.9) 

® 

Lebanon 

449 (3.7) 

® 

Malaysia 

440 (5.4) 

® 

1 Georgia 

431 (3.8) 

® 

Thailand 

427 (4.3) 

® 

^ Macedonia, Rep. of 

426 (5.2) 

® 

Tunisia 

425 (2.8) 

® 

Chile 

416 (2.6) 

® 

^ Iran, Islamic Rep. of 

415 (4.3) 

® 

v Qatar 

410 (3.1) 

® 

^ Bahrain 

409 (2.0) 

® 

^ Jordan 

406 (3.7) 

® 

^ Palestinian Nat'l Auth. 

404 (3.5) 

® 

^ Saudi Arabia 

394 (4.6) 

® 

v Indonesia 

386 (4.3) 

® 

^ Syrian Arab Republic 

380 (4.5) 

® 

>K Morocco 

371 (2.0) 

® 

^ Oman 

366 (2.8) 

® 

w Ghana 

331 (4.3) 

® 


Mathematics Achievement Distribution 


—r~ 

300 


Country 


Average 
Scale Score 


Mathematics Achievement Distribution 


Ninth Grade Participants 


v Botswana 

397 (2.5) 

® 

1 

■ 

>K South Africa 

552 (2.5) 

® 



2 * Honduras 

338 (3.7) 

® 




Benchmarking Participants 

' z Massachusetts, US 561 (5.3) O 

' Minnesota, US 545 (4.6) O 

i 3 North Carolina, US 537 (6.8) O 

Quebec, Canada 532 (2.3) O 

1 2 Indiana, US 522 (5.1) O 

1 Colorado, US 518 (4.9) O 

1 2 Connecticut, US 518 (4.8) O 

1 2 Florida, US 513 (6.4) O 

2 Ontario, Canada 512 (2.5) O 

2 Alberta, Canada 505 (2.6) 

1 2 California, US 493 (4.9) 

Dubai, UAE 478 (2.1) ® 

i Alabama, US 4«6 (5.9) ® 

Abu Dhabi, UAE 449 (3.7) ® 



O Country average significantly higher than 
the centerpoint of the TIMSS 8th grade scale 
( 3 ) Country average significantly lower than 
the centerpoint of the TIMSS 8th grade scale 


Percentiles of Performance 

5th 25th 75th 95th 


95% Confidence Interval for Average (±2SE) 


>K Average achievement not reliably measured because the percentage of students with achievement too low for estimation exceeds 25%. 

Reservations about reliability of average achievement because the percentage of students with achievement too low for estimation does not exceed 25% but exceeds 1 5%. 
See Appendix C.3 for target population coverage notes 1, 2, and 3. See Appendix C.9 for sampling guidelines and sampling participation notes t, and I. 

( ) Standard errors appear In parentheses. Because of rounding some results may appear Inconsistent. 
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Exhibit 2.4 shows the completed analysis module for this example analysis, and Exhibit 2.5 
displays the results with our four example countries. 

Exhibit 2.4: IEA IDB Analyzer Setup for Example Student-level Analysis with Achievement 


F IEA IDB Analyzer Analysis Module - (Version 3.0.47) 


O Analysis File: | C:\TIMSS2011\Data\BSGALLM5.sav 


0 

Analysis Type: Statistic Type: Plausible Value Option: Benchmark Option: Missing Data Option: Number of Decimals: 

| T1MSS (Using Student Weights) ’ ] [ Percentages and Means w | | Use PVs | yj | [2 w j 


Q Select Variables: 




O Output Files: | C:\TlMSS2011\Data\MATOverall* 


| Modify | 
0 | Start SPSS | 


J | Help 


Return to Main Menu 


TIMSS (Using Student Weights) Percentages and Means Use PVs 


Exhibit 2.5: Output for Example Student-level Analysis with Achievement 


Average for 

BSMMATO by 

IDCNTRY 








N of 

Sum of 


Percent 

BSMMATO 

BSMMATO 


Std . Dev . 

♦COUNTRY ID* 

Cases 

TOTWGT 

Percent 

(s . e. ) 

(Mean) 

(s . e. ) 

Std.Dev 

(s .e. ) 

Australia 

7556 

251985 

1.18 

. 02 

504 . 80 

5 . 09 

85.42 

3.36 

Bahrain 

4640 

12625 

. 06 

. 00 

409.22 

1 . 96 

99.57 

1 . 72 

Armenia 

5846 

36181 

. 17 

. 00 

466.59 

2 . 73 

90 . 68 

1 . 73 

Chile 

5835 

251180 

1.18 

. 03 

416.27 

2.59 

79 . 65 

1 . 85 


Each country’s results are presented on a single line. The countries are identified in the first 
column and the second column reports the number of valid cases. The third column identifies the 
sum of weights of the sampled students. The next four columns report the percentage of students in 
each category (Country) and its standard error, followed by the estimated average mathematics 
achievement and its standard error. The last two columns report the standard deviation of the 
achievement scores and its standard error. 
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As shown in the first line in Exhibit 2.5, Australia had valid data for 7,556 students and that these 
sampled students represented a population of 25 1,985 students. The average mathematics 
achievement in Australia was 504.80 (standard error of 5.09), and its standard deviation was 85.42 
(standard error of 3.36). 

Student-level Analysis with Achievement by Gender 

In our second example, we want to replicate another set of results presented in TIMSS 2011 
International Results in Mathematics. We are interested in investigating the relationship between 
eighth grade student gender and mathematics achievement. These results, presented in Exhibit 1.11 
of TIMSS 2011 International Results in Mathematics, are repeated here in Exhibit 2.6. Because the 
results in this exhibit are based on plausible values, we must ensure that these values are included 
when creating the input file, and also indicate that this analysis will make use of achievement scores. 

After reviewing the appropriate codebook, we observe that the variable ITSEX contains 
categorical information on the gender of students, and this variable is found in the student 
background data files. The Percentages and Means statistic type and the Use PVs plausible value 
option will compute the percentages and average achievement based on plausible values and their 
respective standard errors. The analysis module of the IEA IDB Analyzer is used to perform the 
analysis using the following steps: 

1 . Open the Analysis Module of the IEA IDB Analyzer. 

2. Select the merged data file BSGALLM5 as the Analysis File by clicking the Select button. 

3. Select TIMSS (Using Student Weights) as the Analysis Type. 

4. Select Percentages and Means as the Statistic Type. 

5. Select Use PVs as the Plausible Value Option. 

6. Specify the variable ITSEX as a second grouping variable and activate it by clicking the 
Grouping Variables field. Select ITSEX from the list of available variables and move it to the 
Grouping Variables field by clicking the right arrow ( ► ) button in this section. 

7. Specify the achievement scores to be used for the analysis and activate them by clicking the 
Plausible Values field. Select BSMMAT01-05 from the list of available variables and move it to 
the Plausible Values field by clicking the right arrow ( ► ) button in this section. 

8. The Weight Variable is selected automatically by the software; TOTWGT is selected by default 
because this example analysis uses student background data. 

9. Specify the name and the folder of the output files in the Output Files field by clicking the 
Deline/Modify button. 

10. Click the Start SPSS button to create the SPSS syntax file and open it in an SPSS syntax 
window. The syntax file can be executed by opening the Run menu of SPSS and selecting the 
All menu option. If necessary, the IEA IDB Analyzer will display a prompt to confirm the 
overwriting of existing files. 
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Exhibit 2.6: Exhibit of Example Student-level Analysis with Achievement by Gender, Taken from 

TIMSS 201 7 International Results in Mathematics (Exhibit 1.11) 


Exhibit 1.11 Average Mathematics Achievement by Gender 


TIMSS 2011 
Mathematics 


8 th 

Grade 


Country 

Girls 

Percent of Average Scale 
Students Score 

Boys 

Percent of Average Scale 
Students Score 

Difference 

(Absolute 

Value) 

Gender Difference 
Girls Boys 

Scored Higher Scored Higher 

>K Morocco 

47 (0.8) 

371 (2.3) 

53 (0.8) 

371 (2.7) 

0(3.2) 









2 Russian Federation 

49 (0.9) 

539 (3.8) 

51 (0.9) 

539 (3.9) 

1 (2.91 









Kazakhstan 

49 (0.8) 

486 (4.1) 

51 (0.8) 

488 (4.5) 

2 (3.3) 




1 




Norway 

49 (0.7) 

476 (2.9) 

51 (0.7) 

473 (2.9) 

3 (3.1) 




1 




* England 

48 (2.0) 

508 (5.7) 

52(2.0) 

505 (6.6) 

3 (5.6) 




1 





1 Georgia 

47 (0.9) 

430 (4.1) 

53 (0.9) 

432 (4.4) 

3 (4.0) 





i 




Ukraine 

50 (1.0) 

478 (4.0) 

50(1.0) 

481 (4.9) 

3 (4.4) 









2 United States 

51 (0.6) 

508 (2.9) 

49 (0.6) 

511 (2.8) 

4(2.2) 





■ 




Sweden 

48 (0.9) 

486 (2.1) 

52 (0.9) 

482 (2.4) 

4(2.4) 




1 





Finland 

48 (1.1) 

516(2.7) 

52(1.1) 

512 (2.7) 

4(2.3) 




1 





Slovenia 

49 (0.9) 

502 (2.4) 

51 (0.9) 

507 (2.8) 

5(2.8) 





■ 




Hungary 

49(1.1) 

502 (3.9) 

51 (1.1) 

508 (3.9) 

6(3.5) 





■ 




Hong Kong SAR 

49(1.6) 

588 (5.0) 

51 (1.6) 

583 (4.3) 

6(5.5) 




■ 





Chinese Taipei 

48(1.0) 

613 (3.7) 

52(1.0) 

606 (3.8) 

6(4.1) 




■ 





Korea, Rep. of 

52 (2.5) 

610 (3.5) 

48(2.5) 

616(3.1) 

6(3.1) 





■ 




Iran, Islamic Rep. of 

46 (2.3) 

411 (5.9) 

54(2.3) 

418 (5.9) 

7 (8.1) 





■ 




^ Macedonia, Rep. of 

49 (0.9) 

430 (5.8) 

51 (0.9) 

423 (5.6) 

7(4.7) 




■ 





Japan 

49(1.1) 

566 (3.1) 

51 (1.1) 

574 (3.5) 

8(4.11 





■ 




3 Israel 

50 (1.6) 

520 (3.9) 

50(1.6) 

512(5.2) 

8(4.4) 




■ 





2 Singapore 

49 (0.7) 

615 (3.7) 

51 (0.7) 

607 (4.5) 

9(3.5) 




■ 





Turkey 

49 (0.7) 

457 (3.8) 

51 (0.7) 

448 (4.7) 

9(3.5) 




■ 





Australia 

50 (1.6) 

500 (4.7) 

50 (1.6) 

509 (7.3) 

9(6.9) 









1 Lithuania 

49 (0.7) 

507 (2.6) 

51 (0.7) 

498 (3.2) 

9(3.0) 




■ 





Armenia 

49 (0.8) 

472 (3.1) 

51 (0.8) 

462 (3.2) 

10(3.1) 








^ Syrian Arab Republic 

50 0.7) 

375 (5.3) 

50(1.7) 

385 (5.3) 

11(5.7) 




■■ 




Italy 

49 (0.9) 

493 (2.9) 

51 (0.9) 

504 (2.8) 

11(2.9) 








Romania 

48 (0.9) 

464 (4.6) 

52 (0.9) 

453 (4.2) 

11 (3.6) 








v Qatar 

50 (3.3) 

415 (5.8) 

50 (3.3) 

404 (5.5) 

11(9.51 



■i 





Lebanon 

55 (1.9) 

444 (4.2) 

45(1.9) 

456 (4.7) 

12(4.7) 




■■ 




Indonesia 

50 0.2) 

392 (4.9) 

50(1.2) 

379 (4.5) 

13 (4.0) 








Chile 

53 0.5) 

409 (3.2) 

47(1.5) 

424 (3.0) 

14(3.6) 




■■ 




^ Saudi Arabia 

48 02) 

401 (4.1) 

52(1.2) 

387 (8.0) 

15 (8.9) 








Tunisia 

52 (0.7) 

417 (3.1) 

48 (0.7) 

433 (3.1) 

17(2.5) 








United Arab Emirates 

50 0.7) 

464 (2.7) 

50(1.7) 

447 (3.1) 

17(4.2) 








Thailand 

55 0.6) 

435 (4.2) 

45 (1.6) 

417 (5.3) 

18 (4.4) 








New Zealand 

47 (2.0) 

478 (5.5) 

53 (2.0) 

496 (6.2) 

18(4.7) 








Malaysia 

51 (1.2) 

449 (5.2) 

49(1.2) 

430 (6.2) 

19 (4.4) 








^ Palestinian Nat'l Auth. 

52 (1.7) 

415 (4.2) 

48(1.7) 

392 (5.6) 

23 (7.0) 








* Ghana 

47 (0.8) 

318 (4.8) 

53 (0.8) 

342 (4.3) 

23 (2.91 







^ Jordan 

49(1.7) 

420 (4.3) 

51 (1.7) 

392 (5.9) 

28(7.4) 







Bahrain 

50 (0.8) 

431 (2.5) 

50 (0.8) 

388 (3.1) 

43 (4.0) 







^ Oman 

51 (2.1) 

397 (3.1) 

49(2.1) 

334 (3.8) 

63 (4.6) 









■Mum 













80 

1 

40 

0 

40 

8 


Girls 

Boys 

Difference 

Gender Difference 

Country 

Percent of 

Average Scale 

Percent of 

Average Scale 

(Absolute 

Girls 



Boys 


Students 

Score 

Students 

Score 

Value) 

Scored Higher 

Scored Higher 


Ninth Grade Participants 


>K South Africa 

48(1.0) 

354 (3.0) 

52(1.0) 

350 (3.4) 

3 (4.0) 

1 

^ Botswana 

51 (0.6) 

403 (2.9) 

49 (0.6) 

390 (3.0) 

14(3.1) 

■■ 

2 * Honduras 

55 (1.0) 

328 (4.1) 

45 (1.0) 

351 (4.1) 

23 (3.5) 



Benchmarking Participants 


2 Ontario, Canada 49(0.9) 512(2.7) 51(0.9) 512(3.1) 0(3.1) 








Quebec, Canada 51 (1.4) 531 (2.9) 49 (1.4) 532 (2.5) 0 (2.7) 








1 Minnesota, US 52 (1.5) 545 (4.9) 48 (1.5) 545 (5.1) 0 (3.9) 








’Alabama, US 51 (1.6) 467(6.3) 49 (1.6) 465 (6.2) 2 (3.9) 



i 





Abu Dhabi, UAE 47 (2.7) 450 (3.9) 53 (2.7) 448 (5.7) 2 (6.4) 



i 





2 Alberta, Canada 50(0.9) 504(3.3) 50 (0.9) 506 (2.7) 2 (3.0) 




i 




’ 3 North Carolina, US 52 (1.0) 535 (6.2) 48 (1.0) 539 (8.3) 3 (5.1) 




■ 




’ 2 California, US 49(1.1) 491 (5.6) 51 (1.1) 494 (5.0) 3 (4.1) 




■ 




’Colorado, US 51 (1.5) 516(5.4) 49 (1.5) 520 (5.0) 4 (3.4) 




■ 




’ 2 Connecticut, US 49(1.0) 520(5.2) 51 (1.0) 516 (5.4) 4 (4.5) 



■ 





’ 2 Massachusetts, US 50 (1.0) 558 (6.0) 50 (1.0) 563 (5.5) 5 (4.5) 



■ 




’ 2 Florida, US 49(1.9) 509(6.6) 51 (1.9) 517 (7.3) 8 (5.5) 



■ 




12 Indiana, US 52 (1.1) 518(5.1) 48 (1.1) 526 (5.9) 8 (4.0) 



■ 




Dubai, UAE 48(4.6) 486(4.3) 52 (4.6) 470(5.4) 16 (8.9) 







1 1 

1 1 1 

1 1 


80 40 0 40 80 

■ Difference statistically significant 

■ Difference not statistically significant 


>K Average achievement not reliably measured because the percentage of students with achievement too low for estimation exceeds 25%. 

V Reservations about reliability of average achievement because the percentage of students with achievement too low for estimation does not exceed 25% but exceeds 1 5%. 
See Appendix C.3 for target population coverage notes 1, 2, and 3. See Appendix C.9 for sampling guidelines and sampling participation notes t, 4, and i. 

( ) Standard errors appear in parentheses. Because of rounding some results may appear inconsistent 
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Exhibit 2.7 shows the completed analysis module for this example analysis and the results are 
presented in Exhibit 2.8. 

Exhibit 2.7: IEA IDB Analyzer Setup for Example Student-level Analysis with Achievement by Gender 


f IEA IDB Analyzer Analysis Module - (Version 3.0.47) 


O Analysis File: | C:\TIMSS2011\Data\BSGALLM5.sav 


0 

Analysis Type: Statistic Type: Plausible Value Option: Benchmark Option: Missing Data Option: Number of Decimals: 

| TIMSS (Using Student Weights) ’ ] [ Percentages and Means w | | Use PVs | yj | [2 w j 


0 Select Variables: 


[ Name 

Description 

F 

<2$ BSDCWKHW 

•WEEKLY TIME SPENT ON CHEMISTRY HOMEWORK’ * 

40BSDPWKHW 

•WEEKLY TIME SPENT ON PHYSICS HOMEWORK’ 

4$ BSDEWKHW 

•WEEKLY TIME SPENT ON EARTH SCI HOMEWORK* 

40DPCDATE 

'RLE CREATION DATE AT DPC HAMBURG’ 

4 IDBOOK 

•ACHIEVEMENT TEST BOOKLET* 

4 IDSCHOOL 

’SCHOOL ID* 

4 IDCLASS 

•CLASS ID’ 

4 IDSTUD 

'STUDENT ID’ 

4 IDGRADE 

’GRADE ID* 

4 ITBIRTHD 

•DATE OF STUDENTS BIRTHXDAY* 

1 4 HBIRTHM 

’DATE OF S1UDEN1S BIRTHXMONTH* 

4 ITB1RTHY 

•DATE OF STUDENTS BIRTHYYEAR' 

4 ITADMIM 

TEST ADMINISTRATORS POSITION’ 

4 ITDATE 

•DATE OF TESTING’ 

4 ITLANG 

•LANGUAGE OF TESTING’ 

4 ITACCOMMl 

•SPECIAL ACCOMMODATION/ACH SESSION* 

4 IDPOP 

•POPULATION ID’ 

4 IDG RADER 

•STANDARDIZED GRADE ID’ 

4 IDSTRATE 

’EXPLICIT STRATUM CODE* G 

4 IDSTRAT! 

•IMPLICIT STRATUM CODE* 



0 Output Files: | C:\7IMSS2011\Data\MATbyGender.’ 


| Modify | 
O [ Start SPSS | 


Return to Main Menu 


TIMSS (Using Student Weights) Percentages and Means Use PVs 


Exhibit 2.8: Output for Example Student-level Analysis with Achievement Scores by Gender 


Average for BSMMATO by IDCNTRY ITSEX 


♦COUNTRY ID* 

*SEX OF 
STUDENTS* 

N of 
Cases 

Sum of 
TOTWGT 

Percent 

Percent 
(s.e. ) 

BSMMATO 

(Mean) 

BSMMATO 

(s.e.) 

Std . Dev 

Std . Dev . 
(s.e.) 

Australia 

GIRL 

3747 

125555 

49.83 

1.61 

500.41 

4 . 72 

82 . 72 

3.59 


BOY 

3809 

126429 

50 . 17 

1.61 

509.16 

7.26 

87.80 

4 . 82 

Bahrain 

GIRL 

2288 

6276 

49.72 

. 85 

430 . 78 

2.51 

87.23 

1.93 


BOY 

2352 

6348 

50.28 

. 85 

387 . 89 

3 . 07 

106.20 

2.26 

Armenia 

GIRL 

2894 

17702 

48 . 93 

. 78 

471.52 

3 . 07 

87.13 

1.81 


BOY 

2952 

18479 

51.07 

. 78 

461 . 86 

3.21 

93 . 72 

2.24 

Chile 

GIRL 

3133 

133160 

53 . 01 

1.46 

409.46 

3.23 

79 . 97 

2.39 


BOY 

2702 

118020 

46.99 

1.46 

423 . 94 

3 . 05 

78.59 

2 . 03 


Each country’s results are displayed on two lines, one for each value of the ITSEX variable. The 
countries are identified in the first column and the second column describes the category of ITSEX 
being reported. The third column reports the number of valid cases and the fourth the sum of weights 
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of the sampled students. The next four columns report the percentage of students in each category 
and its standard error, followed by the estimated average mathematics achievement and its standard 
error. The last two columns report the standard deviation of the achievement scores and its standard 
error. From the first two lines of results, 49.83 percent of students in Australia were girls, and 50. 17 
percent were boys. The average mathematics achievement was 500.41 for girls (standard error 
of 4.72) and 509.16 for boys (standard error of 7.26). 

Student-level Regression Analysis 

This example is an extension of the previous example (Student-level Analysis with Achievement by 
Gender), where we will examine the difference in mathematics achievement between girls and boys 
using regression and determine if it is statistically significant. The results of this example also are 
presented in Exhibit 1.11 of TIMSS 2011 International Results in Mathematics and shown in 
Exhibit 2.6, in the column labeled “Difference.” 

For this example, the values of the variable ITSEX are recoded into variable REGSEX by 
running the special SPSS syntax file SYNTAX BSGALLM5.SPS shown in Exhibit 2.9 and provided 
in the International Database. 

Exhibit 2.9: Example SPSS Program to Recode Variables for Student-level Regression Analysis 


* Compute new variable REGSEX from ITSEX . 

get file = " <datpath>\BSGALLM5 . sav" . 

compute REGSEX = ITSEX - 1 . 

value labels 

REGSEX 0 "GIRL" 

1 "BOY" . 

variable labels 

REGSEX "Recoded ITSEX (Girls = 0; Boys = 1) 
save outfile = "<datpath>\BSGALLM5 . sav" . 


The parameter <datpath> in the SPSS syntax shown in Exhibit 2.9 needs to be edited to specify 
the location of the input and output data files. 

REGSEX has a value of zero for girls and one for boys. By using REGSEX, the regression 
intercept (or constant) will be the estimated average mathematics achievement of girls, whereas the 
regression slope (called REGSEX in the SPSS output) will be the estimated increase in the average 
mathematics achievement of boys. This example of a regression analysis is performed by the 
analysis module of the IEA IDB Analyzer using the following steps: 

1 . Open the Analysis Module of the IEA IDB Analyzer. 

2. Specify the data file BSGALLM5 as the Analysis File by clicking the Select button, after having 
run the SPSS syntax file SYNTAX_BSGALLM5.SPS to create the variable REGSEX. 

3. Select TIMSS (Using Student Weights) as the Analysis Type. 


3 This form of variable recoding, known as “dummy coding,” makes the interpretation of regression coefficients easier. It essentially transforms a 

regression analysis into an analysis of variance to test for differences among groups. 
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4. Select Regression as the Statistic Type. 

5. Select Use PVs as the Plausible Value Option. 

6. The variable IDCNTRY is selected automatically as Grouping Variables. No additional 
grouping variables are needed for this analysis. 

7. Click the Non-Plausible Values field in the Independent Variables section to activate it and 
select the variable REGSEX as the analysis variable. This is done by selecting REGSEX from the 
list of available variables and moving it to the Non-Plausible Values field by clicking the right 
arrow ( ► ) button in this section. 

8. Click the Plausible Values button in the Dependent Variable section and click the Plausible 
Values field to activate it. Select BSMMAT01-05 as the analysis variable from the list of 
available variables and move it to the Plausible Values field by clicking the right arrow ( ► ) 
button in this section. 

9. The Weight Variable is selected automatically by the software; TOTWGT is selected by default 
because this example analysis uses student background data. 

10. Specify the name and the folder of the output files in the Output Files field by clicking the 
Deline/Modify button. 

11. Click the Start SPSS button to create the SPSS syntax file and open it in an SPSS syntax 
window. The syntax file will be executed by opening the Run menu of SPSS and selecting the 
All menu option. If necessary, the IEA IDB Analyzer will display a prompt to confirm the 
overwriting of existing files. 

Exhibit 2.10 shows the completed analysis module for this example analysis, and Exhibit 2.1 1 
displays the results. 

Each country’s results are displayed on two lines: the first for the intercept (CONSTANT), and 
the second for the REGSEX coefficient. Generally, there will be as many lines per country as there 
are regression coefficients including the intercept. The first line of results in Exhibit 2.1 1, labeled 
“(CONSTANT),” is the estimated average mathematics achievement of eighth grade girls in 
Armenia, which was 47 1 .52 with a standard error of 3.07. This estimate concurs with the results 
obtained in the previous example (Exhibit 2.8). The eighth grade boys in Armenia had a negative 
average achievement difference of -9.66. With an estimated standard error of 3.10, this achievement 
difference is statistically significant at the 95% confidence level. Adding the two regression 
coefficients together (471.52 - 9.66) yields the estimated average achievement of eighth grade boys 
in Annenia, which was 461.86, the same estimate from the previous example. As a note of interest, 
the IEA IDB Analyzer also computes standardized regression coefficients, whereby the independent 
variables are standardized to have a mean of zero and standard deviation of one (1). 
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Exhibit 2.1 0: IDB Analyzer Setup for Example Student-level Regression Analysis 


£ IEA IDB Analyzer; Analysis Module - (Version 3.0.47) 


O Analysis File: C:\TTMSS2011\Data\BSGALLM55av 


a 

Analysis Type: 

Statistic Type: 

Plausible Value Option: Benchmark Option: 

Missing Data Option: Number of Decimals: 


| TIMSS (Using Student Weights) 

w | | Regression 

H | Use PVs 

| Listwise ’ 2 ▼ | 








0 Select Variables: 



Name 

Description 


iBSSSG01-05 

1ST TO 5TH PLAUSIBLE VALUE SCIENCE* 

X- BSMALG01-05 

1ST TO 5THPV ALGEBRA- 

X- BSMDAT01-05 

IST TO 5TH PV DATA AND CHANCE* 

X. BSMNUM01-05 

1ST TO 5THPV NUMBER* 

X~ BSMGEO01-05 

1ST TO 5TH PV GEOMETRY* 

X. BSSCHE01-05 

1ST TO 5TH PV CHEMISTRY* 

X- BSSEAR01-05 

1ST TO 5TH PV EARTH SCIENCE* 

X- BSSBIO01-05 

1ST TO 5TH PV BIOLOGY* 

X- BSSPHY01-05 

1ST TO 5TH PV PHYSICS* 

X- BSMKNO01-05 

1ST TO 5TH PV MATH KNOWING’ 

X. BSMAPP01-05 

1ST TO 5TH PV MATH APPLYING* 

X- BSMREA01-05 

1ST TO 5TH PV MATH REASONING* 

iBSSKNO01-05 

1ST TO 5TH PV SCIENCE KNOWING* 

X- BSSAPP01-05 

1ST TO 5TH PV SCIENCE APPLYING* 

X- BSSREA01-05 

1ST TO 5TH PV SCIENCE REASONING* 




O Output Files: j C:\TIMSS2011\Data\MATGefKierDiff.* 


| Modify j 
J | Start SPSS ] 


Return to Main Menu 


TIMSS (Using Student Weights) Regression Use PVs 


Exhibit 2.1 1 : Output for Example Student-level Regression Analysis 


Regression Coefficients 


IDCNTRY 

Variable 

Regression 

Coefficient 

Regression 
Coefficient 
(s .e . ) 

Regression 

Coefficient 

(t-value) 

Stndrdzd . 
Coefficient 

Stndrdzd . 
Coefficient 
(s .e. ) 

Stndrdzd . 
Coefficient 
(t-value) 

Armenia 

(CONSTANT) 

471.52 

3 . 07 

153 . 75 





REGS EX 

-9 . 66 

3 . 10 

-3 . 12 

- . 05 

. 02 

-3.11 

Australia 

(CONSTANT) 

500.41 

4 . 72 

105.93 





REGS EX 

8 . 75 

6 . 90 

1.27 

. 05 

. 04 

1.28 

Bahrain 

(CONSTANT) 

430 . 78 

2.51 

171.50 





REGSEX 

-42 . 89 

3 . 99 

-10 . 74 

- .22 

. 02 

-11.16 

Chile 

(CONSTANT) 

409.46 

3.23 

126 . 86 





REGSEX 

14 .48 

3 . 63 

3 . 99 

.09 

. 02 

3 . 88 


Percentages of Students Reaching the TIMSS International Benchmarks 

This section describes how to use the IEA IDB Analyzer to perform analyses of student achievement 
in relation to the TIMSS International Benchmarks. As an example, we will compute the percentages 
of TIMSS 2011 students reaching the four TIMSS International Benchmarks of eighth grade 
mathematics achievement — Advanced, High, Intermediate, and Low — using the merged 
BSGALLM5 data fde. These results, presented in Exhibit 2.19 of TIMSS 2011 International Results 
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in Mathematics, are shown here in Exhibit 2.12. This example is performed by the analysis module 
of the IEA IDB Analyzer using the following steps: 

1 . Open the Analysis Module of the IEA IDB Analyzer. 

2. Specify the data fde BSGALLM5 as the Analysis File by clicking the Select button. 

3. Select TIMSS (Using Student Weights) as the Analysis Type. 

4. Select Benchmarks as the Statistic Type. 

5. Select the Cumulative option under the Benchmark Option drop-down menu to get cumulated 
percentages of students reaching the TIMSS international benchmarks. 

6. The variable IDCNTRY is selected automatically as Grouping Variables. No additional 
grouping variables are needed for this analysis. 

7. Specify the achievement scores to be used for the analysis by clicking the Plausible Values field. 
Select BSMMAT01-05 from the list of available variables and move it to the Plausible Values 
field by clicking the right arrow ( ► ) button in this section. 

8. Specify the TIMSS International Benchmarks — 400, 475, 550, and 625 (Low, Intermediate, 

High, and Advanced, respectively). Enter these four values in the Achievement Benchmarks 
field, each separated by a blank space. 

9. The Weight Variable is selected automatically by the software; TOTWGT is selected by default 
because this example analysis uses student background data. 

10. Specify the name and the folder of the output files in the Output Files field by clicking the 
Deline/Modify button. 

11. Click the Start SPSS button to create the SPSS syntax file and open it in an SPSS syntax 
window. The syntax file will be executed by opening the Run menu of SPSS and selecting the 
All menu option. If necessary, the IEA IDB Analyzer will display a prompt to confirm the 
overwriting of existing files. 

Exhibit 2.13 shows the completed analysis module for this example analysis, and Exhibit 2.14 
presents the results. 

As shown in the first few lines of results, 89. 17 percent of the eighth grade students in Australia 
perfonned at or above the Low International Benchmark of 400, with a standard error of 1.08; 

62.94 percent reached the Intermediate International Benchmark, with a standard error of 2.40; 28.65 
percent reached the High International Benchmark, with a standard error of 2.63; and 8.68 percent of 
the students reached the Advanced International Benchmark, with a standard error of 1.68. 
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Exhibit 2.1 2: Example Exhibit of International Benchmark Analysis, Taken from 
TIMSS 2011 International Results in Mathematics (Exhibit 2.19) 


Exhibit 2.19: Performance at the international Benchmarks of 
Mathematics Achievement 


Country 


Chinese Taipei 

2 Singapore 
Korea, Rep. of 
Hong Kong SAR 
Japan 

2 Russian Federation 

3 Israel 
Australia 

* England 
Hungary 
Turkey 

2 United States 


Percentages of Students Reaching 
International Benchmarks 


• Advanced 
O High 

• Intermediate 
O Low 



• 

0 

• 0 


2 

0 





_ J 


# ■ o ■ ■ ♦ o 


Romania 
1 Lithuania 
New Zealand 
Ukraine 
Slovenia 
Finland 
Italy 


Armenia 


Kazakhstan 
^ Macedonia, Rep. of 
' Georgia 

United Arab Emirates 
v Qatar 

^ Iran, Islamic Rep. of 
Malaysia 
Thailand 
v Bahrain 
Sweden 

^ Palestinian Nat'l Auth. 
Lebanon 
Norway 


-O 
— O 


-O 
— O 


v Saudi Arabia 
Chile 


^ Oman 


Tunisia 

•-O- 

v Syrian Arab Republic 

•O- 

v Indonesia 

•o — 

* Morocco 

•o— 

* Ghana 



TIMSS 2011 
Mathematics 


Advanced 

High 

Intermediate 

Low 

Benchmark 

Benchmark 

Benchmark 

Benchmark 

(625) 

(550) 

(475) 

(400) 

49 (1.5) 

73 (1.0) 

88 (0.7) 

96 (0.4) 

48 (2.0) 

78 (1.8) 

92 (1.1) 

99 (0.3) 

47 (1.6) 

77 (0.9) 

93 (0.6) 

99 (0.2) 

34 (2.0) 

71 (1.7) 

89 (1.4) 

97 (0.8) 

27(1.3) 

61 (1.3) 

87 (0.7) 

97 (0.3) 

14(1.2) 

47 (2.0) 

78 (1.4) 

95 (0.7) 

12 (1.2) 

40 (1.7) 

68 (1.8) 

87 (1.2) 

9(1.7) 

29(2.6) 

63 (2.4) 

89 (1.1) 

8(1.4) 

32 (2.9) 

65 (2.7) 

88 (1.6) 

8(0.7) 

32 (1.4) 

65 (1.6) 

88 (1.2) 

7 (0.9) 

20(1.2) 

40 (1.5) 

67 (1.3) 

7 (0.8) 

30 (1.4) 

68 (1.3) 

92 (0.7) 

5 (0.8) 

19(1.3) 

44 (1.7) 

71 (1.5) 

5 (0.6) 

29(1.3) 

64 (1.4) 

90 (0.7) 

5 (0.8) 

24 (2.6) 

57 (2.8) 

84 (1.6) 

5 (0.6) 

22 (1.6) 

53 (2.0) 

81 (1.4) 

4 (0.4) 

27(1.2) 

67 (1.4) 

93 (0.7) 

4(0.5) 

30(1.5) 

73 (1.5) 

96 (0.6) 

3 (0.5) 

24(1.1) 

64 (1.4) 

90(1.1) 

3 (0.4) 

18 (0.9) 

49 (1.4) 

76 (1.2) 

3 (0.7) 

23 (1.8) 

57 (2.1) 

85 (1.3) 

3 (0.6) 

12 (1.3) 

35 (1.9) 

61 (1.9) 

3 (0.3) 

13 (1.0) 

36 (1.5) 

62 (1.6) 

2 (0.2) 

14 (0.7) 

42 (1.1) 

73 (0.9) 

2 (0.3) 

10 (0.8) 

29 (1.2) 

54 (1.4) 

2(0.5) 

8(1.1) 

26 (1.6) 

55 (1.8) 

2 (0.4) 

12 (1.5) 

36 (2.4) 

65 (2.5) 

2 (0.4) 

8(1.3) 

28 (1.9) 

62 (2.1) 

1 (0.2) 

8(0.7) 

26 (0.7) 

53 (0.8) 

1 (0.3) 

16 (0.9) 

57 (1.1) 

89 (0.7) 

1 (0.3) 

7 (0.7) 

25 (1.3) 

52 (1.5) 

1 (0.2) 

9(1.0) 

38 (2.2) 

73 (1.9) 

1 (0.2) 

12 (0.9) 

51 (1.6) 

87 (1.3) 

1(0.2) 

5 (0.8) 

20 (1.7) 

47 (2.0) 

1 (0.2) 

5 (0.6) 

23 (1.1) 

57 (1.6) 

0(0.1) 

6(0.5) 

26 (1.2) 

55 (1.7) 

0(0.1) 

4(0.3) 

16 (0.6) 

39 (1.1) 

0(0.2) 

5 (0.9) 

25 (1.4) 

61 (1.3) 

0(0.1) 

3 (0.5) 

17 (1.4) 

43 (1.9) 

0(0.1) 

2 (0.5) 

15 (1.2) 

43 (2.1) 

0 (0.0) 

2 (0.2) 

12 (0.5) 

36 (1.0) 

0 (0.0) 

1 (0.2) 

5 (0.8) 

21 (1.8) 


International Median 


Country 

Percentages of Students Reaching 
International Benchmarks 

• Advanced 
O High 

• Intermediate 
O Low 

Advanced 

Benchmark 

(625) 

High 

Benchmark 

(550) 

Intermediate 

Benchmark 

(475) 

Low 

Benchmark 

(400) 


Ninth Grade Participants 


^ South Africa 


O 


1 (0.1) 

3 (0.4) 

9 (0.7) 

24 (1.0) 

v Botswana 

•0 


0 

0(0.1) 

2 (0.5) 

15 (1.0) 

50 (1.4) 

2 * Honduras 

©• 

— O 


0 (0.0) 

1 (0.3) 

4 (0.9) 

21 (1.7) 


Benchmarking Participants 

1 2 Massachusetts, US + O • O 19 (3.0) 57 (3.2) 88 (1.4) 98 (0.3) 


1 3 North Carolina, US 


0 

• O 14 (2.6) 

44 (3.6) 

78 (2.5) 

95 (1.3) 

1 Minnesota, US 

•— 

o 

• O 13 (2.3) 

49 (2.8) 

83 (1.9) 

97 (0.7) 

1 2 Connecticut, US 

» - 


- ■ ■« 10(1.3) 

37 (2.9) 

69 (2.5) 

91 (1.4) 

i 2 Florida, US 

’ • 


— m O 8 (1.6) 

31 (3.2) 

68 (3.3) 

94 (1.3) 

1 Colorado, US 

• 

o 

• O 8(1.1) 

35(2.7) 

71 (2.5) 

93 (1.1) 

1 2 Indiana, US 

• 

o 

• ® 7(1.2) 

35 (3.3) 

74(2.3) 

95 (1.0) 

Quebec, Canada 

— • 

0 

• O 6 (0.6) 

40 (1.8) 

82 (1.3) 

98 (0.4) 

Dubai, UAE 

■ # 


O 5 (0.7) 

23 (1.2) 

53 (1.0) 

79 (0.8) 

1 2 California, US 

— • 

— o r 

O 5 (0.9) 

24(2.5) 

59 (2.8) 

87 (1.7) 

2 Ontario, Canada 

-• 

o 

• O 4 (0.6) 

31 (1.4) 

71 (1.4) 

94 (0.7) 

2 Alberta, Canada 

-• 

— o 

• O 3 (0.5) 

24(1.3) 

69 (1.6) 

95 (0.7) 

1 Alabama, US 

• O- 


■ O 2 (0.8) 

15(2.5) 

46 (3.1) 

79 (2.2) 

Abu Dhabi, UAE 

O 


2 (0.5) 

12 (1.2) 

39 (1.8) 

71 (1.5) 


0 

25 50 

75 100 





>K Average achievement not reliably measured because the percentage of students with achievement too low for estimation exceeds 25%. 

V Reservations about reliability of average achievement because the percentage of students with achievement too low for estimation Is less than 25% but exceeds 1 5%. 
See Appendix C.3 for target population coverage notes 1, 2, and 3. See Appendix C.9 for sampling guidelines and sampling participation notes t, t, and I. 

( ) Standard errors appear In parentheses. Because of rounding some results may appear Inconsistent. 
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Exhibit 2.1 3: IDB Analyzer Setup for Example International Benchmark Analysis 


F IEA IDB Analyzer Analysis Module - (Version 3.0.47) 


O Analysis File: C:\TIMSS2011\Data\BSGALLM5iav 


o 





Analysis Type: 

Statistic Type: 

Plausible Value Option: Benchmark Option: 

Missing Data Option: Number of Decimals: 


| TIMSS (Using Student Weights) 

* | | Benchmarks 

’ | Use PVs | Cumulative 

•1 |2 -1 








£ Select Variables: 


N 

— "d — -~ tj 





X BSSSG01-05 

1ST TO 5TH PLAUSIBLE VALUE SCIENCE’ 

;* 

XBSMALG01-05 

1ST TO 5THPV ALGEBRA’ 


X BSMDAT01-05 

1ST TO 5TH PV DATA AND CHANCE’ 


X BSMNUM01-05 

1STT0 5THPV NUMBER* 


X BSMGEO01-05 

1ST TO 5THPV GEOMETRY’ 


XBSSCHE01-05 

1ST TO 5TH PV CHEMISTRY* 


X BSSEAR01-05 

1ST TO 5TH PV EARTH SCIENCE* 


X BSSBIO01-05 

1ST TO 5TH PV BIOLOGY* 


X BSSPHY01-05 

1ST TO 5THPV PHYSICS’ 


X BSMKN 001-05 

1ST TO 5TH PV MATH KNOWING’ 


X BSMAPP01-05 

1ST TO 5TH PV MATH APPLYING’ 


X BSMREA01-05 

1ST TO 5TH PV MATH REASONING’ 


XBSSKNO01-05 

1ST TO 5TH PV SCIENCE KNOWING* 


X BSSAPP01-05 

1ST TO 5TH PV SCIENCE APPLYING* 


X BSSREA01-05 

1ST TO 5TH PV SCIENCE REASONING* 






Output Files: | C:\71MSS2011\Data\MATIBM.* 


| Modify | 
O I Start SPSS | 


Return to Main Menu 


TIMSS (Using Student Weights) Benchmarks Use PVs Cumulative 


Exhibit 2.1 4: Output for Example International Benchmark Analysis 


Percent reaching 

benchmarks (400 475 

550 625) 

Of BSMMAT0 




Performance 


N of 

Sum of 


Percent 

♦COUNTRY ID* 

Group 



Cases 

TOTWGT 

Percent 

(s . e. ) 

Australia 

At 

or 

Above 

400 

6540 

224703 

89.17 

1.08 


At 

or 

Above 

475 

4479 

158602 

62 . 94 

2.40 


At 

or 

Above 

550 

1980 

72193 

28 . 65 

2 . 63 


At 

or 

Above 

625 

525 

21868 

8 . 68 

1.68 

Bahrain 

At 

or 

Above 

400 

2603 

6753 

53.49 

.79 


At 

or 

Above 

475 

1314 

3307 

26 . 19 

. 65 


At 

or 

Above 

550 

415 

1007 

7.97 

.68 


At 

or 

Above 

625 

85 

160 

1.26 

.25 

Armenia 

At 

or 

Above 

400 

4588 

27636 

76.38 

1.16 


At 

or 

Above 

475 

2994 

17735 

49.02 

1.37 


At 

or 

Above 

550 

1123 

6386 

17.65 

.88 


At 

or 

Above 

625 

221 

1167 

3.23 

.40 

Chile 

At 

or 

Above 

400 

3654 

142826 

56 . 86 

1.57 


At 

or 

Above 

475 

1772 

57652 

22 . 95 

1.11 


At 

or 

Above 

550 

541 

13437 

5.35 

.62 


At 

or 

Above 

625 

66 

1399 

.56 

. 16 
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Student-level Analysis with a Contextual Scale 

TIMSS 2011 has innovated in its reporting of contextual data by creating contextual scales based on 
Rasch modeling. 4 A number of contextual scales were reported in the TIMSS 2011 International 
Reports and are available in the International Database for analysis. Each contextual scale variable is 
a Rasch score with an international mean of 10 and an international standard deviation of 2. An index 
was derived from each scale that divides the range of scores into usually three categories: the most 
desirable scores (high values), the least desirable scores (low values), and the remaining scores in 
between. 

These contextual scales and their corresponding indices were reported in the TIMSS 2011 
International Reports. Exhibit 2.15 shows one such example, Exhibit 8.2 from TIMSS 2011 
International Results in Mathematics, reporting how much students like learning mathematics. 
Results on the Rasch scale are reported for each country as an “Average Scale Score” and its 
corresponding index is reported as the percentages of students in each category — Like Learning 
Mathematics, Somewhat Like Learning Mathematics, and Do Not Like Learning Mathematics — 
along with the average achievement in each category. The Rasch scores for this contextual scale are 
contained in the BSBGSLM variable; the corresponding index is found in the BSDGSLM variable. 

This example will replicate both the average scale score of the Students Like Learning 
Mathematics contextual scale and the percentages of students, with their average mathematics 
achievement, in each category of its index. This will be done in two steps, both using the merged 
BSGALLM5 data fde. 

The first step will compute the average scale score using the contextual scale variable 
BSBGSLM. It is performed by the analysis module of the IEA IDB Analyzer using the following 
steps: 

1 . Open the Analysis Module of the IEA IDB Analyzer. 

2. Specify the data file BSGALLM5 as the Analysis File. 

3. Select TIMSS (Using Student Weights) as the Analysis Type. 

4. Select Percentages and Means as the Statistic Type. 

5. Select None Used as the Plausible Value Option, because we will not use any achievement 
scores for this part of the analysis. 

6. Specify the variable BSBGSLM as the analysis variable and activate it by clicking the Analysis 
Variables field. Select BSBGSLM from the list of available variables and move it to the 
Analysis Variables field by clicking the right arrow ( ► ) button in this section. 

7. The Weight Variable is selected automatically by the software; TOTWGT is selected by default 
because this example analysis uses student background data. 

8. Specify the name and the folder of the output files in the Output Files field by clicking the 
Deline/Modify button. 


4 The contextual scales are described in the context questionnaire scales section of Methods and Procedures in TIMSS and PIRLS 2011. 
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Exhibit 2.15: Example Exhibit of a Contextual Scale, Taken from 

TIMSS 201 7 International Results in Mathematics (Exhibit 8.2) 


Exhibit 8.2: Students Like Learning Mathematics 


TIMSS 2011 

Mathematics 


8 th 


Grade 


Reported by Students 

Students were scored according to their degree of agreement with five statements on the Students Like Learning Mathematics scale. Students who 5 

Like Learning Mathematics had a score on the scale of at least 11.3, which corresponds to their "agreeing a lot" with three of the five statements and £ 

"agreeing a little" with the other two, on average. Students who Do Not Like Learning Mathematics had a score no higher than 9.0, which corresponds | 

to their "disagreeing a little" with three of the five statements and "agreeing a little" with the other two, on average. All other students Somewhat Like >. 

Learning Mathematics. | 

Like Learning Somewhat Like Do Not Like g 

Countrv Mathematics Learning Mathematics Learning Mathematics Average 



Percent 

Average 

Percent 

Average 


Percent 

Average 


Scale Score 

Si 


of Students 

Achievement 

of Students 

Achievement 


of Students 

Achievement 



I 

Morocco 

48 (0.7) 

398 (2.4) 

40 (0.7) 

353 (2.2) 


12 (0.5) 

340 (4.6) 


11.2 (0.03) 

T 

Armenia 

43 (1.0) 

499 (3.1) 

39 (0.8) 

451 (3.4) 

■1 

18 (1.0) 

437 (4.8) 

■ ■ 

10.9 (0.05) 

5 

Jordan 

42 (IS) 

442 (3.7) 

39 (1.0) 

388 (4.2) 


19 (0.9) 

376 (4.8) 


10.9 (0.06) 

| 

Georgia 

42 (1.3) 

463 (5.0) 

40 (1.0) 

423 (4.1) 

■1 

18 (1.0) 

405 (6.2) 

10.8 (0.06) 

5 

Malaysia 

39 (1.3) 

463 (5.0) 

46 (0.9) 

430 (5.6) 


15 (0.9) 

413 (8.1) 


10.8 (0.05) 

1 

Iran, Islamic Rep. of 

39(1.1) 

450 (5.4) 

40 (0.8) 

3% (4.2) 

■1 

22 (0.9) 

388 (4.5) 

■ ■ 

10.6 (0.05) 

c 

Ghana 

38 (1.4) 

370 (4.8) 

51 (1.2) 

314 (4.0) 


10 (0.5) 

299 (6.7) 


10.9 (0.05) 

-C 

£ 

Oman 

38 (0.8) 

420 (3.0) 

45 (0.8) 

342 (3.6) 

■1 

17 (0.7) 

324 (4.4) 

■ ■ 

10.8 (0.04) 

Kazakhstan 

38 (1.5) 

506 (4.4) 

52 (1.3) 

478 (4.4) 


10 (0.7) 

475 (7.4) 


10.9 (0.05) 

1 

Tunisia 

38 (1.0) 

448 (3.4) 

40 (0.8) 

415 (3.2) 


23 (0.9) 

405 (3.3) 


10.6 (0.05) 

3 

Syrian Arab Republic 

37(1.1) 

408 (5.2) 

44 (1.0) 

373 (4.8) 


19 (0.9) 

353 (6.3) 


10.7 (0.05) 

z> 

Z 

Ukraine 

36(1.7) 

502 (4.9) 

43 (1.2) 

477 (4.1) 


20 (1.2) 

450 (4.9) 


10.6 (0.07) 


Lebanon 

35 (1.2) 

475 (4.6) 

43 (1.0) 

441 (4.2) 


21 (1.1) 

425 (5.6) 


10.6 (0.06) 


Singapore 

32 (0.7) 

637 (3.9) 

44 (0.7) 

610 (4.1) 

■1 

23 (0.7) 

578 (4.4) 

■ ■ 

10.4 (0.03) 


Turkey 

31 (1.0) 

504 (6.0) 

42 (0.7) 

436 (3.9) 


26 (1.0) 

420 (3.5) 


10.3 (0.05) 


United Arab Emirates 

31 (0.7) 

488 (2.3) 

42 (0.6) 

448 (2.5) 


27 (0.8) 

432 (2.5) 


10.2 (0.04) 


Palestinian Nat'l Auth. 

31 (1.1) 

447 (5.0) 

43 (1.0) 

394 (4.1) 


26 (1.1) 

375 (5.1) 


10.3 (0.05) 


Russian Federation 

29(1.1) 

567 (4.7) 

49 (0.9) 

537 (3.6) 

■1 

22 (1.0) 

509 (4.1) 

■ ■ 

10.3 (0.04) 


Saudi Arabia 

29 (1.3) 

436 (5.6) 

40 (1.0) 

389 (5.4) 


32 (1.4) 

364 (4.1) 


10.1 (0.07) 


Qatar 

27 (1.0) 

456 (4.5) 

43 (0.8) 

401 (3.7) 

m 

31 (1.2) 

386 (4.8) 

■ ■ 

10.0 (0.05) 


Thailand 

26 (1.1) 

456 (5.6) 

57 (0.9) 

420 (4.5) 


16 (1.0) 

408 (5.1) 


10.3 (0.05) 


Israel 

26 (0.8) 

536 (5.1) 

40 (0.7) 

523 (4.3) 


35 (1.0) 

496 (5.1) 


9.9 (0.04) 


Macedonia, Rep. of 

24 (1.0) 

462 (6.2) 

40 (1.0) 

422 (6.3) 


36 (1.4) 

425 (6.0) 


9.8 (0.06) 


Bahrain 

24 (0.6) 

454 (4.6) 

38 (0.9) 

413 (2.7) 

■■ 

38 (0.8) 

381 (3.4) 

■ ■ 

9.8 (0.03) 


Lithuania 

22 (1.0) 

531 (3.7) 

44 (1.0) 

506 (2.7) 


34(1.1) 

482 (3.3) 


9.8 (0.05) 


Chile 

22 (0.9) 

449(3.5) 

40 (0.9) 

416 (2.9) 

■■ 

38 (1.0) 

398 (2.9) 

■ ■ 

9.8 (0.04) 


Indonesia 

20 (1.4) 

396 (6.1) 

70 (1.2) 

385 (4.5) 


10 (0.8) 

382 (6.3) 


10.4 (0.04) 


United States 

19 (0.6) 

536 (3.2) 

40 (0.6) 

515 (3.0) 


40 (0.8) 

494 (2.8) 

■ ■ 

9.5 (0.04) 


Hong Kong SAR 

19 (0.8) 

635 (4.4) 

44 (1.0) 

595 (3.8) 


37(1.3) 

551 (4.6) 


9.6 (0.05) 


Romania 

18(1.0) 

516 (6.1) 

40 (1.0) 

459 (4.3) 

■■ 

41 (1.2) 

438 (4.8) 

■ ■ 

9.5 (0.05) 


Italy 

18 (0.9) 

538 (3.6) 

42 (0.9) 

507 (2.8) 


40 (1.3) 

472 (3.2) 


9.6 (0.05) 


New Zealand 

17(1.0) 

525 (6.9) 

41 (1.0) 

497 (5.7) 

■■ 

42 (1.5) 

467 (4.8) 

■ ■ 

9.5 (0.06) 


Norway 

17 (0.9) 

511 (4.1) 

42 (1.0) 

482 (2.6) 


42 (1.4) 

453 (2.8) 


9.4 (0.05) 


Australia 

16 (0.9) 

553 (7.5) 

40 (0.9) 

520(5.6) 

45 (1.4) 

476 (4.4) 


9.3 (0.06) 


Hungary 

15 (0.7) 

549 (5.6) 

35 (1.0) 

508 (4.8) 


50(1.3) 

491 (3.8) 


9.2 (0.05) 


England 

14(1.0) 

548 (8.9) 

44 (1.3) 

517 (5.7) 

■■ 

42 (1.7) 

484 (5.2) 

■ ■ 

9.4 (0.07) 


Chinese Taipei 

14 (0.7) 

681 (4.3) 

33 (0.9) 

645 (3.6) 


53 (1.2) 

568 (3.2) 


9.0 (0.06) 


Sweden 

13 (0.6) 

524 (4.0) 

42 (0.7) 

498 (1.8) 


44(1.0) 

462 (2.1) 


9.4 (0.04) 


Finland 

10 (0.6) 

560 (4.1) 

34 (1.0) 

532 (2.8) 


57 (1.1) 

496 (2.6) 


8.8 (0.05) 


Japan 

9(0.6) 

621 (5.1) 

38 (1.1) 

589 (3.3) 

■■ 

53 (1.4) 

545 (3.1) 

■ I 

9.1 (0.05) 


Korea, Rep. of 

8 (0.3) 

677 (4.7) 

36 (0.7) 

649 (3.3) 


56 (0.8) 

581 (2.9) 


8.9 (0.03) 


Slovenia 

6(0.4) 

544 (5.3) 

31 (1.1) 

521 (3.0) 


63 (1.3) 

494 (2.4) 

■ ■ 

8.6 (0.05) 


International Avg. 

26(0.2) 

504 (0.8) 

42 (0.1) 

467 (0.6) 

_L 

31 (0.2) 

443 (0.7) 

■ 




Ninth Grade Participants 


Botswana 

47(1.1) 

427 (2.5) 

38 (0.8) 

376 (2.6) 

16 (0.8) 

370 (4.3) 

11.0(0.05) 

South Africa 

41 (0.9) 

378 (2.0) 

44 (0.7) 

339 (2.9) 

15 (0.6) 

348 (5.3) 

10.8 (0.04) 

Honduras 

23 (1.0) 

364 (4.6) 

49 (0.9) 

332 (4.2) 

28(1.1) 

334 (4.8) 

10.1 (0.06) 


Benchmarking Participants 


Abu Dhabi, UAE 

32 (U) 

485 (4.4) 

42 (1.0) 

441 (3.6) 

26 (1.4) 

420 (4.9) 


10.3 (0.06) 

Dubai, UAE 

29 (1.0) 

508 (3.5) 

41 (0.9) 

473 (3.1) 

30 (1.0) 

456 (3.1) 

1 ■ 

10.1 (0.05) 

Ontario, Canada 

26(1.1) 

546 (3.5) 

41 (1.0) 

513 (3.4) 

34 (1.4) 

481 (3.0) 


9.9 (0.06) 

North Carolina, US 

24(1.8) 

556 (7.6) 

44(1.1) 

542 (7.8) 

31 (2.3) 

516 (7.0) 

1 ■ 

9.9(0.11) 

Connecticut, US 

22 (IS) 

552 (6.0) 

40 (1.2) 

526 (5.2) 

38 (1.8) 

495 (5.4) 


9.7 (0.08) 

Colorado, US 

20 (1.6) 

548 (5.9) 

38(1.7) 

528 (4.8) 

42 (2.1) 

495 (5.8) 


9.4 (0.10) 

Massachusetts, US 

19(1.3) 

585 (6.1) 

40 (1.0) 

568 (5.4) 

41 (1.7) 

543 (5.4) 


9.4 (0.09) 

Minnesota, US 

18 (li) 

578 (6.8) 

41 (0.9) 

555 (4.7) 

41 (1.6) 

521 (4.6) 

1 ■ 

9.5 (0.08) 

Alabama, US 

18 (1.9) 

475 (10.7) 

37 (0.9) 

471 (6.7) 

45 (1.7) 

460 (5.3) 


9.3(0.11) 

Florida, US 

17(1.1) 

552 (9.7) 

38(1.4) 

525 (6.9) 

45 (1.7) 

493 (6.2) 

1 ■ 

9.4 (0.08) 

California, US 

17 (0.9) 

519 (6.4) 

42 (1.3) 

4% (6.1) 

41 (1.8) 

480 (5.0) 


9.4 (0.07) 

Alberta, Canada 

16 (0.9) 

531 (4.7) 

44 (1.0) 

514 (2.5) 

40 (1.4) 

486 (3.1) 

1 ■ 

9.4 (0.06) 

Indiana, US 

16 (1.4) 

547 (6.2) 

39(1.3) 

529 (5.3) 

45 (2.0) 

507 (5.0) 


9.3 (0.10) 

Quebec, Canada 

12 (0.7) 

557 (3.9) 

43 (0.9) 

540 (2.4) 

44 (1.2) 

517 (2.6) 

1 ■ 

9.3 (0.05) 


Centerpolnt of scale set at 1 0. 

( ) Standard errors appear in parentheses. Because of rounding some results may appear Inconsistent. 
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9. Click the Start SPSS button to create the SPSS syntax file and open it in an SPSS syntax 
window. The syntax file can be executed by opening the Run menu of SPSS and selecting the 
All menu option. If necessary, the IEA IDB Analyzer will display a prompt to confirm the 
overwriting of existing files. 

Exhibit 2.16 shows the completed analysis module for this example analysis, and Exhibit 2.17 
displays the results. 

Exhibit 2.1 6: IDB Analyzer Setup for Example Student-level Analysis with a Contextual Scale (Step 1 ) 


f IEA IDB Analyzer Analysis Module - (Version 3.0.47) 




O Analysis File: C:\TlMSS2011\Data\BSGALLM5iav 


G 

Analysis Type: 

Statistic Type: 

Plausible Value Option: Benchmark Option: 

Missing Data Option: Number of Decimals: 


| TIMSS (Using Student Weights) 

T | Percentages and Means T 

| None Used ’ | 

U 








O Select Variables: 


[ Name 

Description 


BSMIBM05 

'INTERN. MATH BENCH REACHED WITH 5TH PV* * 

BSSIBM01 

•INTERN. SO BENCH REACHED WITH 1ST PV* 

ESSIBM02 

•INTERN. SO BENCH REACHED WITH 2ND PV 

40BSSIBMO3 

•INTERN. SO BENCH REACHED WITH 3RD PV* 

BSSIBM04 

•INTERN. SO BENCH REACHED WITH 4TH PV* 

40BSSIBMO5 

INTERN. SO BENCH REACHED WITH 5TH PV* 

4$BSBGHER 

•HOME EDUCATIONAL RESOURCES /SCL* 

40BSDGHER 

•HOME EDUCATIONAL RESOURCES/IDX* 

^ BSBGSBS 

•STUDENTS BULLIED AT SCHOOl/SCL* 

<$ BSDGSBS 

•STUDENTS BULLIED AT SCHOOL7IDX’ 

BSDGSLM 

•STUDENTS LIKE LEARNING MATHEMATICS/IDX’ 

BSEGSLS 

•STUDENTS LIKE LEARNING SCIENCE/SCL’ 

BSDGSLS 

•STUDENTS LIKE LEARNING SCIENCE/IDX’ 

BSBGSLB 

•STUDENTS LIKE LEARNING BIOLOGY/SCL* 

4$BSDGSLB 

•STUDENTS UKE LEARNING BIOLOGY/IDX* Cl 

BSBGSLC 

•STUDENTS UKE LEARNING CHEMISTRY/SCL’ 

BSDGSLC 

•STUDENTS UKE LEARNING CHEMISTRY/IDX’ 

BSBGSLP 

’STUDENTS UKE LEARNING PHYSICS/SCL* 

BSDGSLP 

’STUDENTS UKE LEARNING PHYSICS/IDX* 

^Ibsbgsle 

•STUDENTS UKE LEARNING EARTH SCIENCE/SCL* T 



J Output Files: | C:\TlMSS2011\Data\BSBGSLM.* 


| Modify | 
O | Start SPSS | 


Return to Main Menu 


TIMSS (Using Student Weights) Percentages and Means None Used 


Exhibit 2.17: Output for Example Student-level Analysis with a Contextual Scale (Step 1 ) 


Average for 

BSBGSLM by 

(IDCNTRY) 









N of 

Sum of 


Percent 

BSBGSLM 

BSBGSLM 


Std.Dev. 

Percent 

♦COUNTRY ID* 

Cases 

TOTWGT 

Percent 

(s.e. ) 

(Mean) 

(s.e. ) 

Std.Dev. 

(s.e. ) 

Missing 

Australia 

7389 

248762 

45.70 

.74 

9.32 

. 06 

1.91 

. 03 

1.28 

Bahrain 

4581 

12458 

2.29 

. 03 

9.77 

. 03 

2 . 05 

. 02 

1.32 

Armenia 

5626 

34775 

6.39 

. 14 

10 . 87 

. 05 

1 . 92 

. 02 

3 . 89 

Chile 

5772 

248321 

45.62 

.77 

9.76 

. 04 

1.96 

. 03 

1 . 14 


As shown in the first line of the results, students in Australia scored 9.32 on the Students Like 
Learning Mathematics contextual scale, with a standard error of 0.06. Note that this is below the 
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international average of 10, the difference being statistically significant at the 95% confidence level. 
The IEA IDB Analyzer also computes the standard deviation of the contextual scale and its standard 
error. 

In the second step, we will compute the percentages of students, with their average mathematics 
achievement, in each category of the index variable BSDGSLM. It is performed by the analysis 
module of the IEA IDB Analyzer using the following steps: 

1 . Open the Analysis Module of the IEA IDB Analyzer. 

2. Specify the data file BSGALLM5 as the Analysis File. 

3. Select TIMSS (Using Student Weights) as the Analysis Type. 

4. Select Percentages and Means as the Statistic Type. 

5. Select Use PVs as the Plausible Value Option, because we will be computing average 
mathematics achievement by the grouping variable BSDGSLM. 

6. Specify the variable BSDGSLM as a second grouping variable by clicking the Grouping 
Variables field to activate it. Select BSDGSLM from the list of available variables and move it 
to the Grouping Variables field by clicking the right arrow ( ► ) button in this section. 

7. Specify the achievement scores to be used for the analysis by clicking the Plausible Values field 
to activate it. Select BSMMAT01-05 from the list of available variables and move it to the 
Plausible Values field by clicking the right arrow ( ► ) button in this section. 

8. The Weight Variable is automatically selected by the software; TOTWGT is selected by default 
because this example analysis uses student background data. 

9. Specify the name and the folder of the output files in the Output Files field by clicking the 
Define/Modify button. 

10. Click the Start SPSS button to create the SPSS syntax file and open it in an SPSS syntax 
window. The syntax file can be executed by opening the Run menu of SPSS and selecting the 
All menu option. If necessary, the IEA IDB Analyzer will display a prompt to confirm the 
overwriting of existing files. 

Exhibit 2.18 shows the completed analysis module for this example analysis and the results are 
presented in Exhibit 2.19. 

As shown in the first three lines of the results, 15.67 percent of eighth grade students in Australia 
liked learning mathematics (standard error of 0.94) and their average mathematics achievement 
was 553.21 (standard error of 7.47); 39.81 percent somewhat liked learning mathematics (standard 
error of 0.87) and their average mathematics achievement was 519.53 (standard error of 5.58); and 
44.53 percent of students did not like learning mathematics (standard error of 1.41) and their average 
mathematics achievement was 475.97 (standard error of 4.43). 
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Exhibit 2.1 8: IDB Analyzer Setup for Example Student-level Analysis with a Contextual Scale (Step 2) 


F IEA IDB Analyzer Analysis Module - (Version 3.0.47) 


[ 1—1 1 i°i) bad 


O Analysis File: C:\TIMSS2011\Data\BSGALLM53av 


© 

Analysis Type: 

Statistic Type: 

Plausible Value Option: Benchmark Option: 

Missing Data Option: Number of Decimals: 


| TIMSS (Using Student Weights) 

T | | Percentages and Means w 

[ Use PVs H 

ri -1 








0 Select Variables: 




J Output Files: | C:\71MSS2011\Data\BSDGSLM * 


| Modify | 
0 | Start SPSS | 


J | Help 


Return to Main Menu 


TIMSS (Using Student Weights) Percentages and Means Use PVs 


Exhibit 2.1 9: Output for Example Student-level Analysis with a Contextual Scale (Step 2) 


Average for 
♦COUNTRY ID* 

BSMMATO by IDCNTRY BSDGSLM 

♦STUDENTS LIKE LEARNING 
MATHEMATICS/ IDX* 

N of 
Cases 

Sum of 
TOTWGT 

Percent 

Percent 
(s . e. ) 

BSMMATO 

(Mean) 

BSMMATO 
(s .e. ) 

Std . Dev 

Std. Dev. 
(s .e . ) 

Australia 

LIKE LEARNING MATHEMATICS 

1068 

38977 

15.67 

.94 

553.21 

7.47 

85 . 05 

4 . 87 


SOMEWHAT LIKE LEARNING MATHEMATICS 

2985 

99021 

39.81 

. 87 

519.53 

5.58 

84 . 86 

3 . 87 


DO NOT LIKE LEARNING MATHEMATICS 

3336 

110764 

44.53 

1.41 

475 . 97 

4.43 

74.27 

2 . 84 

Bahrain 

LIKE LEARNING MATHEMATICS 

1072 

2959 

23.75 

.64 

454.13 

4 . 64 

97.58 

3 . 63 


SOMEWHAT LIKE LEARNING MATHEMATICS 

1756 

4780 

38.37 

. 86 

413.45 

2.71 

97.39 

2 . 08 


DO NOT LIKE LEARNING MATHEMATICS 

1753 

4719 

37 . 88 

. 84 

380.58 

3.37 

90 . 05 

2.59 

Armenia 

LIKE LEARNING MATHEMATICS 

2421 

14925 

42 . 92 

.97 

499.36 

3 . 07 

85 . 95 

2.49 


SOMEWHAT LIKE LEARNING MATHEMATICS 

2181 

13729 

39.48 

.76 

451.39 

3.37 

86.66 

1.75 


DO NOT LIKE LEARNING MATHEMATICS 

1024 

6121 

17.60 

.97 

436 . 88 

4 . 75 

83 . 82 

3.33 

Chile 

LIKE LEARNING MATHEMATICS 

1289 

54770 

22 . 06 

. 86 

449.31 

3.52 

81 . 08 

2 . 78 


SOMEWHAT LIKE LEARNING MATHEMATICS 

2291 

99860 

40.21 

.89 

416.42 

2 . 93 

78.34 

2 . 19 


DO NOT LIKE LEARNING MATHEMATICS 

2192 

93691 

37.73 

.97 

398 .30 

2 . 95 

73.56 

1.53 


TIMSS & PIRLS 

International Study Center 

Lynch School of Education, Boston College 


USING THE IEA IDB ANALYZER TO ANALYZE 
THE TIMSS 2011 INTERNATIONAL DATABASE 

CHAPTER 2 29 




Computing Correlations with Background Variables and Achievement Scores 

In addition to the analyses described above, the IEA IDB Analyzer (version 3.0) is able to compute 
correlations among background variables as well as between background variables and achievement 
scores. While these types of analyses will not be demonstrated here, the steps for conducting them 
are the same as those described previously: select the grouping variables, the analysis variables, the 
achievement scores (if necessary), and confirm the weight variable. The output will display, for each 
group defined by the grouping variables, the correlation coefficients for each possible pair of 
variables. When using only background variables, the diagonal and elements above the diagonal of 
the correlation matrix are displayed with their corresponding standard errors. When using 
achievement scores, a single column is displayed containing the correlations between each of the 
background variables specified in the model and the achievement scores selected. 

Calculating Percentiles of Achievement 

The Percentiles statistic type is an additional tool provided by the IEA IDB Analyzer (version 3.0) 
for analyzing the TIMSS 2011 data. This procedure will compute the percentiles of a student 
achievement distribution for a specified set of plausible values and within any specified subgroups of 
students, along with appropriate standard errors. 

2.6 TIMSS Analyses with Teacher-level Data 

Analyses with teacher background data seek to make statements about students whose teachers have 
a given characteristic, rather than about teachers with a given characteristic. As our example of an 
analysis using teacher background data, we will investigate the percentage of eighth grade students 
according to the years of experience of their mathematics teachers. The results of such an analysis 
are presented in Exhibit 7.6 of TIMSS 2011 International Results in Mathematics and are reproduced 
here in Exhibit 2.20. 

We will use the Percentages and Means statistic type and select the Use PVs option to estimate 
the percentages of students, with their average mathematics achievement, by reporting categories of 
teachers’ years of experience. 

As with the previous examples, we first proceed to identify the variables relevant to the analysis 
in the appropriate files, and review the documentation for any specific national adaptations to the 
questions of interest. Because we are using a teacher-level variable, we need to look in the 
mathematics teacher background data files for the variable that contains the infonnation on the years 
of experience of eighth grade mathematics teachers. The variable BTBG01 contains information on 
teachers’ years of experience. That information was collapsed into reporting categories and stored in 
the derived variable BTDG01. 5 

The merged data file BTMALLM5 will be used for this example teacher-level analysis, which 
will be performed by the analysis module of the IEA IDB Analyzer using the following steps: 

1 . Open the Analysis Module of the IEA IDB Analyzer. 

2. Select the merged data file BTMALLM5 as the Analysis File. 


5 


Supplement 3 to this User Guide describes the construction of the teacher-level derived variable BTDG01. 
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Exhibit 2.20: Exhibit of Example Teacher-level Analysis, Taken from 

TIMSS 2011 International Results in Mathematics (Exhibit 7.6) 


Exhibit 7.6: Teachers'Years of Experience 





TIMSS 2011 13] 
Mathematics 

Reported by Teachers 










Country 

20 Years or More 

At Least 1 0 but Less 
than 20 Years 

At Least 5 but Less 
than 10 Years 

Less than 5 Years 

Average 
Years of 
Experience 

Percent of 
Students 

Average 

Achievement 

Percent of 
Students 

Average 

Achievement 

Percent of 
Students 

Average 

Achievement 

Percent of 
Students 

Average 

Achievement 

Armenia 

63 (3.7) 

467 (3.9) 

30 (3.3) 

464 (6.0) 

4 (1.6) 

473 (24.9) 

3 (1.4) 

474 (18.4) 

24 (0.8) 

Australia r 

37 (4.0) 

519 (8.1) 

22 (3.4) 

513 (10.8) 

18 (3.2) 

504 (17.1) 

24 (3.4) 

485 (8.4) 

15(0.9) 

Bahrain 

19 (2.2) 

433 (7.0) 

54 (3.6) 

404 (3.7) 

17(2.7) 

403 (5.8) 

10 (1.9) 

430 (9.1) 

14 (0.4) 

Chile 

49 (3.8) 

415 (4.6) 

15 (2.9) 

416 (10.0) 

13 (2.8) 

421 (12.1) 

22 (3.4) 

421 (6.3) 

19(1.0) 

Chinese Taipei 

24(3.6) 

621 (7.2) 

41 (3.6) 

607 (5.8) 

26 (3.5) 

608 (9.3) 

9 (2.5) 

593 (8.9) 

14 (0.7) 

England 

21 (3.6) 

510(15.5) 

25(3.7) 

516(11.8) 

22 (3.9) 

495 (11.6) 

32 (3.9) 

503 (10.7) 

12 (0.9) 

Finland 

41 (3.4) 

517 (2.8) 

27 (3.4) 

511 (5.3) 

18 (2.8) 

515 (6.1) 

15 (2.4) 

510 (5.2) 

16 (0.7) 

Georgia 

63 (3.9) 

428 (5.2) 

21 (3.5) 

441 (10.1) 

9 (2.4) 

439 (15.0) 

7 (2.3) 

431 (18.5) 

25(1.1) 

Ghana 

6(1.8) 

360 (19.9) 

23 (3.8) 

340 (9.0) 

28 (4.0) 

334 (9.3) 

43 (3.9) 

321 (6.8) 

8(0.5) 

Hong Kong SAR 

18(3.3) 

570 (11.9) 

39(4.3) 

590 (8.4) 

25 (4.2) 

589 (11.9) 

18 (3.3) 

588 (10.1) 

12(0.7) 

Hungary 

62 (3.5) 

508 (4.4) 

26 (3.0) 

508 (6.2) 

7 (1.9) 

488 (18.6) 

5 (1.5) 

456 (21.5) 

22 (0.7) 

Indonesia 

25 (3.9) 

402 (9.1) 

30 (4.0) 

399 (9.1) 

19 (3.3) 

385 (8.0) 

26 (4.5) 

356 (9.1) 

13 (0.8) 

Iran, Islamic Rep. of 

28 (3.2) 

443 (8.9) 

40 (3.8) 

416 (6.0) 

16 (2.6) 

402 (10.4) 

16 (2.8) 

374 (10.7) 

14 (0.6) 

Israel 

38 (2.8) 

545 (6.6) 

36 (2.8) 

518 (6.6) 

15 (2.0) 

495 (10.7) 

11 (1.8) 

468 (14.4) 

17 (0.5) 

Italy 

60(4.1) 

502 (3.2) 

22(3.3) 

492 (7.3) 

11 (2.5) 

504 (9.1) 

8 (2.1) 

492 (13.6) 

22 (0.9) 

Japan 

47 (3.9) 

576 (3.7) 

18(3.1) 

558(5.5) 

17 (2.3) 

575 (9.1) 

18 (3.1) 

559 (7.5) 

17 (0.8) 

Jordan 

16 (2.6) 

406 (8.5) 

29 (3.3) 

410 (7.6) 

29 (3.5) 

394 (9.6) 

26 (3.1) 

413 (7.0) 

11 (0.6) 

Kazakhstan 

62 (3.9) 

492 (5.2) 

21 (3.2) 

468 (8.6) 

9 (2.7) 

489 (14.9) 

8 (2.2) 

493 (14.8) 

22 (0.9) 

Korea, Rep. of 

34(3.1) 

618 (5.0) 

22 (2.8) 

616 (8.8) 

17 (2.1) 

625 (7.1) 

27 (2.6) 

594 (4.8) 

13 (0.6) 

Lebanon 

27 (3.61 

454 (7.9) 

32 (3.9) 

445 (6.9) 

21 (3.2) 

460 (9.8) 

20 (3.5) 

445 (8.7) 

14 (1.0) 

Lithuania 

73 (3.4) 

501 (3.0) 

17 (2.6) 

509 (6.8) 

7 (2.1) 

504 (19.6) 

3 (1.4) 

506 (17.8) 

25 (0.8) 

Macedonia, Rep. of r 

50 (4.4) 

421 (9.1) 

25 (4.2) 

430(12.0) 

12 (2.7) 

415 (15.3) 

13 (2.9) 

420 (18.6) 

20 (0.9) 

Malaysia 

18 (3.0) 

446 (12.2) 

31 (3.4) 

446 (9.5) 

21 (3.0) 

426 (11.4) 

30 (3.3) 

441 (10.5) 

11 (0.7) 

Morocco 

69 (2.8) 

374 (2.8) 

11 (2.0) 

373 (9.0) 

5(1.5) 

358 (12.2) 

15 (2.3) 

363 (6.3) 

22 (0.6) 

New Zealand 

36 (3.0) 

492 (8.4) 

22 (2.7) 

486 (9.6) 

25 (3.0) 

489 (8.9) 

17 (2.8) 

482 (15.6) 

15 (0.8) 

Norway 

30 (4.0) 

478 (3.7) 

25 (3.6) 

474 (5.5) 

19 (3.7) 

475 (4.4) 

26 (3.5) 

474 (4.0) 

15 (1.0) 

Oman 

7(1.3) 

362 (12.2) 

25 (2.6) 

385 (6.5) 

46 (3.3) 

363 (4.7) 

21 (2.6) 

360 (6.9) 

9 (0.3) 

Palestinian Nat'l Auth. 

14(3.1) 

413(11.9) 

37 (3.9) 

410 (7.3) 

24 (3.6) 

400 (7.6) 

25 (3.2) 

394 (7.5) 

11 (0.7) 

Qatar 

23 (4.2) 

432 (12.7) 

36 (4.6) 

425 (9.4) 

25 (3.4) 

388 (9.2) 

16 (2.9) 

386 (10.1) 

13 (0.7) 

Romania 

66 (3.7) 

466 (5.2) 

24 (3.3) 

449 (9.3) 

6 (1.7) 

420 (15.9) 

4(1.6) 

423 (12.7) 

25(0.9) | 

Russian Federation 

67 (3.3) 

540 (4.4) 

24 (3.1) 

543 (7.0) 

5 (1.2) 

515 (15.2) 

4 (1.2) 

547 (23.5) 

24 (0.6) 

Saudi Arabia 

13 (2.9) 

386 (10.2) 

41 (3.9) 

406 (7.3) 

25 (3.5) 

402 (8.9) 

21 (3.5) 

367 (7.7) 

11 (0.6) 

Singapore 

10 (1.4) 

618 (10.6) 

16 (2.1) 

619 (9.3) 

26 (2.4) 

624 (7.3) 

47 (2.5) 

601 (5.0) 

8 (0.4) 

Slovenia 

52 (2.9) 

506 (3.2) 

20 (2.6) 

500 (5.0) 

17 (2.0) 

500 (4.1) 

12 (1.9) 

515 (4.9) 

19 (0.6) 

Sweden r 

26(2.7) 

486 (5.4) 

42 (3.4) 

489 (3.9) 

22 (2.7) 

482 (3.7) 

10 (2.0) 

476 (5.1) 

15(0.6) 

Syrian Arab Republic 

16(3.1) 

400 (9.6) 

26(3.7) 

375 (7.9) 

24 (3.6) 

370 (8.8) 

35 (4.0) 

378 (8.7) 

10 (0.6) 

Thailand 

34 (3.4) 

444 (8.4) 

21 (3.1) 

432 (11.0) 

18 (2.7) 

417(11.6) 

28 (3.2) 

415 (8.7) 

15 (0.8) 

Tunisia 

38(3.3) 

442 (5.6) 

35 (3 J) 

419 (5.4) 

18 (2.8) 

417 (7.5) 

10 (2.1) 

394 (7.2) 

16 (0.7) 

Turkey 

11 (2.2) 

471 (14.5) 

24 (32) 

481 (10.8) 

38 (3.5) 

445 (6.9) 

27 (2.8) 

431 (6.5) 

9 (0.5) 

Ukraine 

68 (4.4) 

477 (4.5) 

20 (3.6) 

491 (10.0) 

9 (2.5) 

473 (11.1) 

3 (1.4) 

473 (18.7) 

25 (1.0) 

United Arab Emirates 

24 (2.0) 

442 (6.4) 

36 (2.4) 

455 (4.0) 

26 (2.3) 

461 (4.8) 

14 (1.8) 

467 (6.8) 

13 (0.4) 

United States r 

26(2.2) 

519 (6.8) 

28 (2.4) 

517(5.1) 

28 (2.8) 

506 (7.2) 

17(2.2) 

505 (6.7) 

14 (0.6) 

i International Avg. 

36(0.5) 

474(1.3) 

28 (0.5) 

470(1.2) 

19 (0.4) 

463 (1.7) 

18 (0.4) 

458 (1.81 

16 (0.1) 1 

Ninth Grade Participants 

Botswana 

2(1.0) 


39 (4.5) 

401 (5.3) 

31 (4.3) 

403 (4.2) 

29 (3.9) 

384 (5.2) 

9 (0.4) 

Honduras r 

26 (3.8) 

341 (6.5) 

23 (4.2) 

335 (10.8) 

22 (4.4) 

332 (8.4) 

29(4.2) 

339 (11.1) 

12 (0.9) 

South Africa 

30 (3.8) 

344 (7.3) 

33 (3.4) 

358 (5.8) 

18 (3.0) 

364 (8.6) 

19(3.1) 

345 (8.7) 

14 (0.8) 

Benchmarking Participants 

Alberta, Canada 

25 (3.5) 

506 (5.0) 

37 (4.3) 

504 (3.8) 

15 (3.0) 

504 (6.9) 

23 (3.4) 

505 (5.3) 

13 (0.7) 

Ontario, Canada 

16 (2.8) 

511 (7.5) 

44 (4.2) 

512 (3.8) 

31 (3.5) 

516 (4.9) 

10(2.5) 

511 (9.4) 

12 (0.5) 

Quebec, Canada 

19 (3.0) 

544 (6.6) 

47 (3.8) 

536 (4.2) 

22 (3.2) 

524 (7.0) 

12 (2.6) 

521 (7.3) 

13 (0.6) 

Abu Dhabi, UAE 

25 (4.1) 

456 (14.3) 

30 (4.1) 

433 (6.3) 

29 (4.6) 

456 (8.5) 

16(32) 

463 (9.2) 

14 (0.9) 

Dubai, UAE 

19 (2.2) 

443 (9.5) 

42 (2.6) 

491 (5.0) 

25 (3.3) 

488 (8.7) 

13 (2.6) 

471 (13.9) 

13 (0.5) 

Alabama, US r 

16 (4.8) 

494 (20.4) 

35 (7.8) 

473 (11.2) 

32 (6.2) 

450 (12.0) 

17 (5.7) 

464(11.2) 

12(1.3) 

California, US r 

19 (5.4) 

502 (25.5) 

33 (6.9) 

490 (9.2) 

28 (6.4) 

506 (10.2) 

20 (5.9) 

479 (21.5) 

12(1.3) 

Colorado, US r 

21 (4.9) 

564 (9.3) 

32 (5.6) 

517(11.3) 

32 (5.8) 

508 (14.0) 

15(3.5) 

471 (13.3) 

13 (1.0) 

Connecticut, US 

29 (6.2) 

531 (17.9) 

32 (5.6) 

533 (9.2) 

20 (4.8) 

509 (18.9) 

19 (5.5) 

503 (14.5) 

14(1.3) 

Florida, US r 

18(5.3) 

530 (13.7) 

43 (7.0) 

521 (10.5) 

26 (5.8) 

514 (14.6) 

13 (4.0) 

524 (29.0) 

13 (1.2) 

Indiana, US r 

34 (5.6) 

526 (11.0) 

22 (5.8) 

533 (13.8) 

27 (6.0) 

516 (12.2) 

17 (5.2) 

494 (9.9) 

15(1.4) 

Massachusetts, US 

10(4.1) 

566 (20.3) 

33 (5.8) 

569 (10.9) 

39 (5.2) 

552 (8.5) 

18 (5.5) 

556 (17.9) 

11 (1.3) 

Minnesota, US 

27 (6.4) 

556 (9.3) 

36 (5.2) 

553 (8.9) 

22 (4.5) 

531 (15.3) 

15 (4.3) 

528 (17.9) 

15(1.5) 

North Carolina, US r 

26 (5.5) 

559 (13.2) 

30 (5.6) 

530 (14.8) 

33 (5.5) 

545 (13.2) 

11 (4.3) 

517 (12.7) 

14(1.0) 


( ) Standard errors appear in parentheses. Because of rounding some results may appear inconsistent. 
A tilde (~) indicates Insufficient data to report achievement. 

An "r" indicates data are available for at least 70% but less than 85% of the students. 
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3. Select TIMSS (Using Math Teacher Weights) as the Analysis Type. 

4. Select Percentages and Means as the Statistic Type. 

5. Select Use PVs as the Plausible Value Option. 

6. Specify the variable BTDG01 as a second grouping variable and activate it by clicking the 
Grouping Variables field. Select BTDG01 from the list of available variables and move it to the 
Grouping Variables field by clicking the right arrow ( ► ) button in this section. 

7. Specify the achievement scores to be used for the analysis and activate them by clicking the 
Plausible Values field. Select BSMMAT01-05 from the list of available variables and move it to 
the Plausible Values field by clicking the right arrow ( ► ) button in this section. 

8. The Weight Variable is selected automatically by the software; MATWGT is selected by default 
because this example analysis uses mathematics teacher background data. 

9. Specify the name and the folder of the output files in the Output Files field by clicking the 
Deline/Modify button. 

10. Click the Start SPSS button to create the SPSS syntax file and open it in an SPSS syntax 
window. The syntax file can be executed by opening the Run menu of SPSS and selecting the 
All menu option. If necessary, the IEA IDB Analyzer will display a prompt to confirm the 
overwriting of existing files. 

Exhibit 2.21 shows the completed analysis module for this example analysis, and Exhibit 2.22 
displays the results. 

Each country’s results are displayed on four lines, one for each value of the BTDG01 variable. 
The results are presented in the same manner as in the previous examples, with countries identified 
in the first column and the second column describing the categories of the analysis variable 
BTDG01. 

As shown in the first four lines of results, 36.54 percent of eighth grade students in Australia 
were taught by teachers with 20 years or more of experience, 21.51 percent were taught by teachers 
with 10 to 19 years of experience, 17.73 percent by teachers with 5 to 9 years of experience, 
and 24.22 percent by teachers with less than 5 years of experience. Also, the estimated average 
mathematics achievement was 519.11 (standard error of 8.07) for eighth grade students taught by 
teachers with 20 years or more of experience, 5 13.05 (standard error of 10.77) for students taught by 
teachers with 10 to 19 years of experience, 503.68 (standard error of 17.14) for students taught by 
teachers with 5 to 9 years of experience, and 485.34 (standard error of 8.45) for students taught by 
teachers with less than 5 years of experience. The IEA IDB Analyzer also produces the standard 
deviations of achievement for all subgroups of BTDG01. 
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Exhibit 2.21 : IDB Analyzer Setup for Example Teacher-level Analysis 


F IEA IDB Analyzer Analysis Module - (Version 3.0.47) 


O Analysis File: | C:\TIMSS2011\Data\BTMALLM5.sav 


o 




Analysis Type: 

Statistic Type: Plausible Value Option: Benchmark Option: 

Missing Data Option: Number of Decimals: 


| TIMSS (Using Math Teacher Weights) 

* | | Percentages and Means » | | Use PVs * \ 

u -i 







0 Select Variables: 




J Output Files: | C:\71MSS2011\Data\MATbyBTDG01.’ 


| Modify | 
O | Start SPSS | 


J | Help 


Return to Main Menu 


TIMSS (Using Math Teacher Weights) Percentages and Means Use PVs 


Exhibit 2.22: Output for Example Teacher-level Analysis 


Average for 
♦COUNTRY ID* 

BSMMATO by IDCNTRY BTDG01 
♦TEACHERS YEARS OF EXPERIENCE* 

N of 
Cases 

Sum of 
MATWGT 

Percent 

Percent 
(s . e. ) 

BSMMATO 

(Mean) 

BSMMATO 
(s . e . ) 

Std.Dev 

Std.Dev. 
(s .e . ) 

Australia 

20 YEARS OR MORE 

2096 

68987 

36.54 

4 . 03 

519.11 

8 . 07 

86.49 

5.53 


AT LEAST 10 BUT LESS THAN 20 YEARS 

1395 

40605 

21.51 

3.39 

513 . 05 

10.77 

79.10 

5.72 


AT LEAST 5 BUT LESS THAN 10 YEARS 

888 

33462 

17.73 

3 . 18 

503.68 

17 . 14 

97.36 

9.44 


LESS THAN 5 YEARS 

1350 

45721 

24.22 

3.38 

485.34 

8.45 

72 . 00 

3.51 

Bahrain 

20 YEARS OR MORE 

966 

2317 

19.25 

2.20 

433 .48 

7 . 02 

104.26 

3 . 65 


AT LEAST 10 BUT LESS THAN 20 YEARS 

2295 

6492 

53.93 

3.56 

404 . 07 

3 . 72 

96.64 

2 . 72 


AT LEAST 5 BUT LESS THAN 10 YEARS 

732 

2077 

17.25 

2.68 

403.36 

5 . 83 

95.74 

3 . 79 


LESS THAN 5 YEARS 

448 

1152 

9.57 

1.93 

429.50 

9 . 09 

98.49 

4 . 97 

Armenia 

20 YEARS OR MORE 

3508 

21506 

62 . 92 

3.71 

466.66 

3 . 90 

91.28 

2 . 13 


AT LEAST 10 BUT LESS THAN 20 YEARS 

1716 

10255 

30 . 00 

3.34 

464.48 

6 . 02 

91.23 

2 . 67 


AT LEAST 5 BUT LESS THAN 10 YEARS 

200 

1423 

4.16 

1.56 

473.15 

24 . 95 

101 . 02 

19 . 86 


LESS THAN 5 YEARS 

134 

999 

2 . 92 

1.40 

474.44 

18.40 

72.17 

8 . 18 

Chile 

20 YEARS OR MORE 

2788 

116439 

49.15 

3 . 81 

415.30 

4.55 

81 . 97 

2 . 69 


AT LEAST 10 BUT LESS THAN 20 YEARS 

878 

35914 

15.16 

2 . 92 

416 . 01 

9.99 

82 . 99 

4 . 07 


AT LEAST 5 BUT LESS THAN 10 YEARS 

689 

31741 

13.40 

2.76 

421.22 

12 . 08 

78 . 09 

4.41 


LESS THAN 5 YEARS 

1166 

52819 

22.29 

3.36 

421.27 

6.30 

74 . 84 

3 . 82 
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2.7 TIMSS Analyses with School-level Data 


When perfonning analyses with school background data, the data are analyzed to make statements 
about students attending schools with a given characteristic, rather than about schools with a given 
characteristic. Our example of an analysis using school background data will compute the 
percentages of eighth grade students, with their average mathematics achievement, who attended 
schools composed of students with different levels of economic background. The results of this 
analysis are presented in Exhibit 5.4 of TIMSS 2011 International Results in Mathematics and are 
replicated here in Exhibit 2.23. 

We will use the Percentages and Means statistic type and select the Use PVs option to estimate 
the percentages of students, with their average mathematics achievement, by reporting categories of 
students’ economic background as reported by school principals. 

The information for this analysis is found in the school-level derived variable BCDG03, where 
schools are characterized as being composed of more affluent students than disadvantaged students, 
composed of more disadvantaged students than affluent students, or composed of neither more 
affluent students nor more disadvantaged students. 6 

The merged data file BCGALLM5 will be used for this example school-level analysis and it is 
performed by the analysis module of the IEA IDB Analyzer using the following steps: 

1 . Open the Analysis Module of the IEA IDB Analyzer. 

2. Select the merged data file BCGALLM5 as the Analysis File. 

3. Select TIMSS (Using Student Weights) as the Analysis Type. 

4. Select Percentages and Means as the Statistic Type. 

5. Select Use PVs as the Plausible Value Option. 

6. Specify the variable BCDG03 as a second grouping variable and activate it by clicking the 
Grouping Variables field. Select BCDG03 from the list of available variables and move it to the 
Grouping Variables field by clicking the right arrow ( ► ) button in this section. 

7. Specify the achievement scores to be used for the analysis and activate it by clicking the 
Plausible Values field. Select BSMMAT01-05 from the list of available variables and move it to 
the Plausible Values field by clicking the right arrow ( ► ) button in this section. 

8. The Weight Variable is selected automatically by the software; TOTWGT is selected by default 
because this example analysis uses school background data linked to student background data. 

9. Specify the name and the folder of the output files in the Output Files field by clicking the 
Deline/Modify button. 

10. Click the Start SPSS button to create the SPSS syntax file and open it in an SPSS syntax 
window. The syntax file can be executed by opening the Run menu of SPSS and selecting the 
All menu option. If necessary, the IEA IDB Analyzer will display a prompt to confirm the 
overwriting of existing files. 


6 


Supplement 3 to this User Guide describes the construction of the school-level derived variable BCDG03 from responses to questions posed to the 
school principals. 
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Exhibit 2.23: Exhibit of Example School-level Analysis, Taken from 

TIMSS 2011 International Results in Mathematics (Exhibit 5.4) 


Exhibit 5.4: School Composition by Student Economic Background 


TIMSS 2011 
Mathematics 


8 th 

Grade 


Reported by Principals 




More Affluent 

• Schools Where More 



More Disadvantaged 

- Schools Where 



than 25% of Students Come from 

Neither More Affluent nor 
More Disadvantaged 

More than 25% of Students Come 



Economically Affluent Homes and Not 

from Economically Disadvantaged 

Country 


More than 25% from Economically 

Homes and Not More than 25% from 



Disadvantaged Homes 



Economically Affluent homes 



Percent 

Average 

Percent 

Average 

Percent 

Average 



of Students 

Achievement 

of Students 

Achievement 

of Students 

Achievement 

Armenia 


35 (3.71 

484 (5.9) 

24 (3.6) 

461 (7.5) 

41 (3.7) 

455 (5.0) 

Australia 

II 

32 (3.41 

543 (11.2) 

39 (3.7) 

507 (6.1) 

29(3.1) 

476 (7.5) 

Bahrain 


45 (0.3) 

420 (3.2) 

28 (0.2) 

408 (2.7) 

27 (0.3) 

395 (3.7) 

Chile 

r 

12(2.3) 

474 (13.0) 

32 (4.1) 

439 (6.0) 

56(3.9) 

399 (4.8) 

Chinese Taipei 


17 (2.7) 

649 (7.9) 

69 (3.8) 

604 (4.2) 

14 (2.9) 

577 (13.5) 

England 

ii 

28 (4.1) 

553 (11.0) 

50 (4.5) 

498 (8.9) 

22 (4.3) 

487 (10.9) 

Finland 

r 

30 (3.4) 

519 (4.0) 

67 (3.8) 

513 (3.0) 

3 (1.5) 

486 (3.5) 

Georgia 


11 (2.0) 

436 (13.7) 

44 (4.4) 

438 (6.8) 

45 (4.2) 

417 (6.8) 

Ghana 


7 (2.0) 

392 (13.9) 

18 (3.4) 

331 (10.6) 

75 (3.6) 

321 (5.2) 

Hong Kong SAR 


11 (3.0) 

628 (11.8) 

37 (5.1) 

609 (10.2) 

53 (4.8) 

561 (7.8) 

Hungary 


16 (2.7) 

535 (7.4) 

33 (4.1) 

531 (4.9) 

50 (4.3) 

478 (5.6) 

Indonesia 

II 

16 (3.3) 

426 (9.9) 

28 (4.6) 

400 (8.1) 

56 (4.6) 

369 (6.0) 

Iran, Islamic Rep. of 


20 (2.7) 

472 (11.2) 

25 (3.5) 

429 (9.1) 

54 (3.8) 

390 (5.2) 

Israel 


28 (3.5) 

556 (7.8) 

30 (4.5) 

526 (8.8) 

42 (3.9) 

481 (8.8) 

Italy 


40 (3.7) 

515 (3.7) 

47 (3.9) 

495 (3.8) 

13 (2.6) 

465 (8.9) 

Japan 

li 

46 (4.4) 

582 (4.5) 

44 (4.5) 

564 (4.1) 

10 (2.9) 

548 (9.0) 

Jordan 

r 

32 (3.5) 

431 (7.0) 

25 (2.9) 

402 (9.7) 

43 (3.9) 

388 (6.3) 

Kazakhstan 


75 (3 J) 

487 (4.4) 

20 (3.4) 

493 (11.0) 

5 (1.8) 

462 (22.5) 

Korea, Rep. of 


18 (3.3) 

653 (5.8) 

51 (4.3) 

612 (2.6) 

32 (3.9) 

591 (4.6) 

Lebanon 

r 

21 (4.1) 

491 (8.8) 

34 (4.2) 

455 (8.7) 

45 (5.0) 

435 (5.3) 

Lithuania 


23 (3.6) 

537 (6.5) 

39 (4.4) 

499 (4.3) 

38 (4.0) 

487 (4.5) 

Macedonia, Rep. of 

r 

38 (3.6) 

458(7.9) 

30 (4.1) 

428 (10.0) 

32 (3.9) 

401 (9.7) 

Malaysia 


26 (3.2) 

467 (10.5) 

23 (3.3) 

452 (12.4) 

52 (4.1) 

424 (8.8) 

Morocco 

r 

6 (1.4) 

422 (15.0) 

13 (2.5) 

393 (9.8) 

81 (2.9) 

361 (2.6) 

New Zealand 


30 (5.6) 

522 (6.9) 

47 (5.8) 

485 (7.4) 

24 (4.0) 

450 (10.6) 

Norway 


— 

-- 

-- 

-- 

- 

- 

Oman 


43 (3.1) 

386 (4.6) 

26 (2.6) 

360 (5.6) 

31 (3.1) 

339 (5.8) 

Palestinian Nat'l Auth. 


44 (4.2) 

411 (6.5) 

23 (3.9) 

402 (8.7) 

33(3.7) 

393 (6.1) 

Qatar 

r 

81 (0.2) 

403 (4.3) 

16 (0.2) 

448 (6.6) 

3 (0.1) 

435 (18.2) 

Romania 


18 (2.9) 

479 (12.7) 

29 (4.2) 

471 (8.1) 

52 (4.3) 

447 (62) 

Russian Federation 


58 (3.5) 

553 (5.1) 

25 (2.8) 

527 (4.4) 

16 (3.1) 

513 (10.3) 

Saudi Arabia 

r 

40 (4.4) 

405 (7.5) 

30 (4.4) 

394 (10.5) 

29 (4.1) 

382 (8.2) 

Singapore 


27 (0.0) 

643 (5.9) 

61 (0.0) 

604 (4.9) 

11 (0.0) 

569(11.6) 

Slovenia 


40 (3.8) 

510 (4.4) 

45 (4.3) 

506 (2.7) 

15 (2.7) 

489 (6.8) 

Sweden 

r 

74 (4.4) 

490 (2.6) 

21 (4.1) 

472 (5.5) 

5(1.8) 

466 (11.9) 

Syrian Arab Republic 

r 

37 (4.2) 

388 (8.0) 

27 (4.3) 

392 (9.5) 

36 (4.4) 

371 (8.2) 

Thailand 


20 (3.0) 

466 (13.9) 

24 (3.6) 

437 (9.5) 

57 (4.4) 

410 (5.7) 

Tunisia 

ll 

23 (3.3) 

439(9.6) 

29 (3.3) 

432 (3.9) 

48 (3 J) 

411 (3.0) 

Turkey 


17 (2.6) 

533 (11.6) 

25 (3.3) 

455 (6.0) 

59 (3.8) 

428 (5.1) 

Ukraine 

li 

13 (2.7) 

486(14.1) 

29 (3.9) 

486 (7.4) 

59 (4.5) 

472 (5.1) 

United Arab Emirates 

r 

70 (2.0) 

459 (3.4) 

17 (1.9) 

442 (7.3) 

13 (1.4) 

441 (5.6) 

United States 


22 (1.9) 

543 (5.8) 

23 (1.9) 

526 (6.1) 

55 (1.9) 

490 (3.4) 

t International Avg. 


32 (0.5) 

494 (1.4) 

33 (0.6) 

471 (1.2) 

36 (OJ) 

448 (1.3) 

Ninth Grade Participants 

Botswana 


13 (3.0) 

432 (10.9) 

24 (4.0) 

401 (4.4) 

63 (4.6) 

384 (2.7) 

Honduras 

S 

5 (1.6) 

383 (12.5) 

14 (3.4) 

358 (12.3) 

82 (3.6) 

333 (4.4) 

South Africa 

r 

8(1.3) 

487 (14.4) 

12 (2.6) 

356 (15.0) 

80 (2.7) 

339 (3.2) 

Benchmarking Participants 

Alberta, Canada 


39 (4.1) 

517 (3.6) 

43 (4.8) 

505 (3.3) 

18 (3.8) 

482 (5.9) 

Ontario, Canada 

li 

37 (4.1) 

523 (5.1) 

36 (4.7) 

510 (3.8) 

27(4.5) 

498 (5.2) 

Quebec, Canada 

r 

51 (4.1) 

542 (4.3) 

32 (3.8) 

523 (5.2) 

17(3.5) 

514 (6.3) 

Abu Dhabi, UAE 

r 

76 (4.1) 

453 (6.1) 

17 (3.6) 

429 (10.3) 

7(2.4) 

446 (14.9) 

Dubai, UAE 

r 

71 (0.3) 

484 (3.2) 

12 (0.2) 

449 (2.9) 

16 (0.2) 

434 (3.8) 

Alabama, US 

r 

17 (4.4) 

492 (19.0) 

5 (3.4) 

481 (41.0) 

78 (5.6) 

455 (6.1) 

California, US 

r 

16 (4.2) 

541 (12.3) 

20 (5.2) 

532 (16.7) 

64 (5.4) 

467 (5.8) 

Colorado, US 

r 

21 (5.7) 

525 (9.1) 

34 (6.6) 

526 (10.9) 

46 (7.4) 

500 (12.5) 

Connecticut, US 

r 

43 (6.1) 

565 (7.8) 

27 (6.1) 

528 (10.3) 

30 (5.9) 

455 (8.6) 

Florida, US 

r 

6(3.4) 

500 (18.4) 

37 (5.6) 

535 (11.1) 

58 (6.0) 

499 (8.8) 

Indiana, US 

r 

13 (4.5) 

573 (7.5) 

29 (5.3) 

524 (10.1) 

58 (5.9) 

509 (6.6) 

Massachusetts, US 

i 

29 (6.8) 

589 (9.1) 

45 (6.6) 

562 (8.0) 

26 (4.2) 

521 (13.4) 

Minnesota, US 


18 (3.2) 

583 (16.6) 

45 (7.1) 

546 (5.4) 

37 (7.6) 

530 (8.4) 

North Carolina, US 

r 

14 (5.6) 

560 (16.1) 

23 (6.4) 

551 (10.9) 

63 (6.7) 

519 (10.5) 


( ) Standard errors appear In parentheses. Because of rounding some results may appear inconsistent. 

A dash (-) indicates comparable data not available. 

An "r'indlcates data are available for at least 70% but less than 85% of the students. An 's’ indicates data are available for at least 50% but less than 70% of the students. 
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SOURCE: lEA's Trends in International Mathematics and Science Study -TIMSS 2011 





Exhibit 2.24 shows the completed analysis module for this example analysis and the results are 
presented in Exhibit 2.25. 

Exhibit 2.24: IDB-Analyzer Setup for Example School-Level Analysis 


& IEA IDB Analyzer Analysis Module - (Version 3 . 0 . 47 ) 


O Analysis File: | C:\TIMSS2011\Data\BCGALLM5.sav | Select | 


o 

Analysis Type: 

Statistic Type: 

Plausible Value Option: Benchmark Option: 

Missing Data Option: Number of Decimals: 


TIMSS (Using Student Weights) 

w Percentages and Means » 

| Use PVs -| 

' I* 








Q Select Variables: 



Name 

Description 

F 

X- BSSSG01-05 

1ST TO 5TH PLAUSIBLE VALUE SCIENCE’ 

JLBSMALG01-05 

1ST TO 5THPV ALGEBRA' 

i_ BSMDAT01-05 

1ST TO 5TH PV DATA AND CHANCE' 

X BSMNUM01-05 

1STT0 5THPV NUMBER' 

X BSMGEO01-05 

1ST TO 5TH PV GEOMETRY’ 

~X_ BSSCHE01-05 

1ST TO 5TH PV CHEMISTRY' 

X. BSSEAR01-05 

1ST TO 5TH PV EARTH SCIENCE* 

XBSSBIO01-05 

1ST TO 5TH PV BIOLOGY* 

X BSSPHY01-05 

1ST TO 5THPV PHYSICS’ 

X BSMKN 001-05 

1ST TO 5TH PV MATH KNOWING’ 

A BSMAPHdl-OS 

1ST TO STH PV MATH APPLYING’ 

X BSMREA01-05 

1ST TO 5TH PV MATH REASONING’ 

X BSSKNO01-05 

1ST TO 5TH PV SCIENCE KNOWING’ 

X BSSAPP01-05 

1ST TO 5TH PV SCIENCE APPLYING' 

X-BSSREA01-05 

1ST TO 5TH PV SCIENCE REASONING' 

1 


A Grouping Variables: SI Exclude Missing From Analysis 


Name Description 


► & 1DCNTRY 'COUNTRY ID* 

#BCDG03 'SCHOOL COMPOSITION BY STUDENT BACKGD* 


yj Analysis Variables: 
A Plausible Values: 



v Dependent Variable: 
Weight Variable: 


Name Description 


> flgTOTWGT TOTAL STUDENT WEIGHT* 

* Achievement Benchmarks / Percentiles: 


J Output Files: | C:\TIMSS2011\Data\MATbyBCDG03 * 


| Modify | 
O | Start SPSS | 


J | Help 


Return to Main Menu 


TIMSS (Using Student Weights) Percentages and Means Use PVs 


Exhibit 2.25: Output for Example School-level Analysis 


Average for 
♦COUNTRY ID* 

BSMMAT0 by IDCNTRY BCDG03 
♦SCHOOL COMPOSITION BY STUDENT 

BACKGD* 

N of 
Cases 

Sum of 
TOTWGT 

Percent 

Percent 
(s.e. ) 

BSMMAT0 

(Mean) 

BSMMAT0 

(s.e.) 

Std.Dev 

Std.Dev 

(s.e.) 

Australia 

MORE AFFLUENT 




2118 

71105 

32.49 

3.36 

542.55 

11 . 18 

89 . 84 

5.18 


NEITHER MORE AFFLUENT 

NOR 

MORE 

DISADVANT 

2535 

84358 

38.54 

3.74 

507.36 

6 . 08 

71.47 

3 .66 


MORE DISADVANTAGED 




1800 

63400 

28 . 97 

3 . 11 

475 . 74 

7.50 

81 . 15 

4 .30 

Bahrain 

MORE AFFLUENT 




1954 

5157 

45.30 

.32 

419.65 

3.19 

104 . 85 

2.68 


NEITHER MORE AFFLUENT 

NOR 

MORE 

DISADVANT 

1143 

3173 

27 . 87 

.23 

407.57 

2.72 

94 . 08 

2 . 88 


MORE DISADVANTAGED 




1051 

3055 

26 . 83 

.34 

395.48 

3.69 

92 .29 

2.29 

Armenia 

MORE AFFLUENT 




2085 

12047 

34.78 

3.70 

483.52 

5 . 89 

88 . 70 

3.54 


NEITHER MORE AFFLUENT 

NOR 

MORE 

DISADVANT 

1329 

8398 

24.25 

3.59 

460.52 

7.53 

90 . 99 

2.63 


MORE DISADVANTAGED 




2158 

14191 

40 . 97 

3.68 

455.43 

5 . 03 

90.49 

2.56 

Chile 

MORE AFFLUENT 




811 

22830 

12 . 16 

2 . 32 

474.35 

13 . 02 

80 . 85 

4.41 


NEITHER MORE AFFLUENT 

NOR 

MORE 

DISADVANT 

1391 

59452 

31.66 

4 . 07 

439.36 

5 . 98 

72 . 78 

3 . 02 


MORE DISADVANTAGED 




2119 

105493 

56 . 18 

3 . 86 

398 . 81 

4.76 

71 . 93 

3 . 08 
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In this example, each country’s results are presented on three lines, one for each value of the 
BCDG03 variable. The results are presented in the same manner as in previous examples, with 
countries identified in the first column and the second column describing the categories of BCDG03. 

As shown in the first three lines of results, 32.49 percent of eighth grade students in Australia 
attended schools with more affluent students than disadvantaged students (standard error of 3.36), 
38.54 percent attended schools with neither more affluent students nor more disadvantaged students 
(standard error of 3.74), and 28.97 percent attended schools with more disadvantaged students than 
affluent students (standard error of 3.1 1). Also, the estimated average mathematics achievement 
was 542.55 (standard error of 1 1.18) for eighth grade students in schools with more affluent students, 
507.36 (standard error of 6.08) for students in schools with neither more affluent students nor more 
disadvantaged students, and 475.74 (standard error of 7.50) for students in schools with more 
disadvantaged students. The IEA IDB Analyzer also produces the standard deviations of 
achievement for all subgroups of BCDG03. 
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Chapter 3 

Using SAS to Analyze the 

TIMSS 201 1 International Database 


3.1 Overview 


This chapter presents some basic examples of analyses that can be perfonned with the TIMSS 2011 
International Database using the SAS statistical analysis system (SAS Institute, 2012) and the SAS 
programs and macros provided. The SAS macros use sampling weights and the jackknifing 
algorithm to deal with the TIMSS complex sample design and take into account plausible values 
when analyzing student achievement. 

Although some familiarity with the structure of the TIMSS 2011 database will be helpful, the 
analyses presented in this chapter are simple in nature and are designed primarily to familiarize users 
with the various data files and their structure, as well as the variables to be used in most analyses. 
Chapter 4 provides a more detailed description of the data files contained in the International 
Database, including their structure and contents, along with detailed information on all the 
supporting documentation provided with the International Database. 

In general, the examples in this chapter compute percentages of students in specified subgroups, 
average mathematics achievement in those subgroups, and appropriate standard errors for these 
statistics. Additional examples compute regression coefficients and their standard errors. The 
example analyses, using student, teacher and school data, replicate some of the analyses that are 
included in the TIMSS 2011 International Reports (Mullis, Martin, Foy, & Arora, 2012; and Martin, 
Mullis, Foy, & Stanco, 2012). Users are encouraged to practice analyzing the TIMSS 2011 data by 
replicating some of the exhibits presented in the TIMSS 2011 International Reports. 

Before conducting any statistical analyses with the TIMSS 2011 International Database, users 
should download and copy the contents of the International Database either on their computer or on a 
server. 1 For the purposes of this chapter, we will assume all files have been copied to the folder titled 
“C:\TIMSS201 1\.” All SAS programs presented in this chapter are available in the International 
Database. They can be adapted to perform a variety of analyses with some basic knowledge of the 
SAS language. With a little experience and some practice with these programs, users should be able 
to make the necessary modifications to obtain the desired results. The example SAS programs invoke 
SAS macros that will be described in this chapter. Although users will be expected to modify the 
example programs, there is no need to make any changes within the SAS macros. 


The TIMSS 201 1 International Database is available on the TIMSS 201 1 International Database and User Guide webpage 
( http://timssandpirls.bc.edu/timss20ll/international-database.html) and on the IEA Study Data Repositoiy website ( http://rms.iea-dpc.org / ). 
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3.2 SAS Programs and Macros 


The TIMSS 2011 International Database includes a number of SAS programs needed to process the 
SAS data fdes, compute survey results, and carry out example analyses. This chapter gives detailed 
instructions on how to adapt and make use of them. The following programs and macros are 
available: 

CONVERT.SAS 

This SAS program converts the SAS export files found in the International Database into SAS data 
files. All programs and macros described in this chapter require that the SAS export files be 
converted into SAS data files. 

ASASCRM5.SAS, BSASCRM5.SAS 

These two SAS programs can be used to convert the response codes to the TIMSS 2011 achievement 
items to their corresponding score levels. 

JOIN.SAS 

This SAS program combines files of the same type from more than one country. 

JACKGEN.SAS (and SAMPLEJACKGEN.SAS) 

This SAS macro program is used to compute weighted percentages of students within defined 
subgroups, along with their means on a specified continuous variable. This macro generates replicate 
weights and computes standard errors using the jackknife repeated replication (JRR) methodology. 
The analysis variable can be any continuous variable. When computing average achievement scores 
with plausible values, the macro JACKPV.SAS should be used. 

JACKPV.SAS (and SAMPLEJACKGPV.SAS) 

This SAS macro program is used to compute weighted percentages of students within defined 
subgroups, along with their average achievement on an achievement scale using the available 
plausible values. This macro generates replicate weights and computes standard errors using the 
jackknife repeated replication (JRR) and multiple imputation methodologies. This macro should be 
used when achievement plausible values are used in an analysis. 

JACKREG.SAS (and SAMPLEJACKREG.SAS) 

This SAS macro program is used to compute regression coefficients and their standard errors within 
defined subgroups. This macro can be used with any analysis variable, but is not appropriate for 
analyzing achievement with plausible values. 

JACKREGP.SAS (and SAMPLEJACKREGP.SAS) 

This SAS macro program is used to compute regression coefficients and their standard errors within 
defined subgroups when using achievement plausible values as the dependent variable. 
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EXAMPLE1.SAS, EXAMPLE2.SAS, EXAMPLE3.SAS, 
EXAMPLE4.SAS, EXAMPLE5.SAS, EXAMPLE6.SAS 


These are the programs used in the example analyses presented in this chapter. 

The four SAS macros— JACKGEN, JACKPV, JACKREG, and JACKREGP— have a 
corresponding sample program that calls the respective macro and prints out the results. These 
sample programs are discussed later in this chapter (See 3.6 SAS Macros to Compute Statistics and 
their Standard Errors). 

3.3 Converting the SAS Export Files 

The TIMSS 2011 International Database provides a program called CONVERT. SAS that converts 
the SAS export files provided in the International Database into SAS data files. This conversion is 
necessary because all of the SAS macros and SAS programs presented in this chapter require the use 
of SAS data files. To convert SAS export files into SAS data files, users should apply the following 
steps: 

1 . Open the SAS program file CONVERT. SAS. 

2. At the beginning of the program, specify the data file type in the parameter “TYPE.” 

3. Specify the path where the SAS export files are located in the parameter “EXPPATH.” 

4. Specify the folder where the converted SAS data files will be located in the parameter 
“DATPATH.” 

5. List all of the countries of interest in the parameter “COUNTRY.” By default, all TIMSS 2011 
countries are listed and the program will automatically select the appropriate list by grade based 
on the file type specified in step 2. 

6. Submit the edited code for processing by SAS. 

An example of the CONVERT program is presented in Exhibit 3.1. This example converts the 
SAS export files of type BSG for all countries. For this example, all SAS export files are located in 
the folder titled “C:\TIMSS201 l\Data\SAS_Data,” where the converted SAS data files also will be 
located. 

Users are advised to run the CONVERT program for all countries and all file types. The file 
types at the fourth grade are ACG, ASA, ASG, ASH, ASR, AST, and ATG. At the eighth grade, the 
file types are BCG, BSA, BSG, BSR, BST, BTM, and BTS. These file types are described in Chapter 
4. In principle, this program needs to be run only once for each file type and should be one of the 
first thing users do with the TIMSS 2011 International Database before undertaking any data 
analyses, particularly the data analysis examples in this User Guide. 
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Exhibit 3.1 : Example of CONVERT Program 


%LET TYPE = BSG ; 

%LET EXP PATH = C:\TIMSS201l\Data\SAS_Data\ ; 

%LET DATPATH = C:\TIMSS201l\Data\SAS_Data\ ; 

% MACRO DOIT ; 

%LET COUNTRY = < List of TIMSS 2011 countries > ; 
%LET 1=1; 

%DO % WHILE ( % LENGTH ( %SCAN (&COUNTRY, &I ) ) ) ; 

%LET CTRY = %SCAN (&COUNTRY, &i) ; 

PROC CIMPORT FILE = " &EXPPATH&TYPE&CTRY . M5 . EXP" 
DATA = "&DATPATH&TYPE&CTRY.M5" ; 

%LET I = %EVAL (&I +1) ; 

%END ; 

%MEND DOIT ; 

%DOIT ; 


3.4 Scores for the TIMSS 2011 Items 

Student achievement in TIMSS 2011 is represented by sets of five plausible values for mathematics 
and science and their content and cognitive domains, and these are the preferred scores for any 
analysis of student achievement. However, analyzing performance on individual items may be of 
interest to some users. Carrying out such analyses may require that the individual items in the 
TIMSS 2011 International Database be assigned their correctness score levels, rather than the actual 
response options selected by students for multiple choice items or the two-digit codes given to 
students’ responses to constructed response items. The International Database provides SAS 
programs to perform this task. 

For multiple choice items, numbers 1 through 4 are used to represent response options A through 
D, respectively, in the TIMSS 2011 achievement data files. These responses must be converted to 
their appropriate score level (“1” for correct and “0” for incorrect) based on each multiple choice 
item’s correct response key. For constructed response items, worth a total of one or two points, two- 
digit codes are used to represent the students’ written responses in the achievement data files. These 
codes also must be recoded to represent the correct point values of the responses — either zero, one, 
or two points. 

For both types of items, special codes are set aside to represent missing data as either “Not 
Administered,” “Omitted,” or “Not Reached.” These special missing codes also must be recoded in 
order to carry out specific item-level analyses. By default, the “Not Administered” response code is 
left as missing and the “Omitted” and “Not Reached” response codes are recoded as incorrect. These 
default settings can be modified within the programs, depending on the requirements of the item- 
level analyses. For example, “Not Reached” responses were treated as missing for the purpose of 
calibrating the TIMSS 2011 items, whereas they were treated as incorrect when deriving 
achievement scores for students. 
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The TIMSS 2011 International Database includes two SAS programs — ASASCRM5.SAS for the 
fourth grade items, and BSASCRM5.SAS for the eighth grade items — which will recode the 
responses to individual items from the achievement data files to their appropriate score levels. To 
score each individual TIMSS 2011 item, the program code in the ASASCRM5 and BSASCRM5 
programs must be adapted by doing the following steps: 

1. Open the SAS program file ASASCRM5.SAS or BSASCRM5.SAS. 

2. Specify the folder where the SAS data files are located in the “LIBNAME” statement. 

3. List all of the countries of interest in the parameter “COUNTRY” (by default, all TIMSS 2011 
countries are listed). 

4. Submit the edited code for processing by SAS. 

Each program uses the student achievement data files as input (ASA/BSA), recodes the 
individual items, and saves the results in SAS data files that have “ASC/BSC” instead of 
“ASA/BSA” as the first three characters in their file names. Exhibit 3.2 shows a condensed version 
of the ASASCRM5 and BSASCRM5 programs to score the individual TIMSS 2011 items. 

Exhibit 3.2: Example of ASASCRM5/BSASCRM5 Programs 


LIBNAME LIBDAT "C : \TIMSS2011\Data\SAS_Data\ " ; 

%LET COUNTRY = < List of TIMSS 2011 countries > ; 

%LET ARIGHT = < List of multiple- choice items where A is correct > ; 

%LET BRIGHT = < List of multiple-choice items where B is correct > ; 

%LET CRIGHT = < List of multiple- choice items where C is correct > ; 

%LET DRIGHT = < List of multiple-choice items where D is correct > ; 

%LET CONSTR = < List of constructed-response items > ; 

% MACRO SCORE IT (ITEM, TYPE, RIGHT, NR, NA, OM, OTHER) ; 


%MEND SCORE IT ; 
% MACRO DOIT ; 


DO 

OVER 

ARIGHT 

%SCOREIT 

(ARIGHT, 

"MC" 

1, 

. R, 

• A, . , 

■ i) 

; END 

DO 

OVER 

BRIGHT 

%SCOREIT 

(BRIGHT, 

"MC" 

2, 

.R, 

• A, . , 

■ i) 

; END 

DO 

OVER 

CRIGHT 

%SCOREIT 

(CRIGHT, 

"MC" 

3, 

.R, 

• A, . , 

■ i) 

; END 

DO 

OVER 

DRIGHT 

%SCOREIT 

(DRIGHT, 

"MC" 

4, 

.R, 

• A, . , 

■ i) 

; END 

DO 

OVER 

CONSTR 

%SCOREIT 

(CONSTR, 

" CR" 

, 

.R, 

• A, . , 

■ i) 

; END 


%MEND DOIT ; 
%DOIT ; 


If “Not Reached” responses are to be treated as missing rather than incorrect, users should 
replace the following statement (which appears twice in the programs): 


IF &ITEM = &NR THEN SCORE = 0 ; 
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with this statement: 


IF &ITEM = &NR THEN SCORE = . ; 


3.5 Joining the TIMSS 201 1 Data Files 

The TIMSS 2011 International Database contains separate data files for each country. The 
International Database provides a SAS program called JOIN.SAS that joins individual country data 
files of a particular type into a single aggregated data file, facilitating joint analyses involving more 
than one country. The JOIN program, however, can only join SAS data files of the same type, and 
thus can be used for the following data file types: ACG/BCG, ASA/BSA, ASC/BSC, ASG/BSG, 
ASH, ASR/BSR, AST/BST, and ATG/BTM/BTS. To create a SAS data file with more than one 
country’s data, users should do the following: 

1 . Open the SAS program file JOIN.SAS. 

2. At the beginning of the program, specify the data file type in the parameter “TYPE.” 

3. Specify the folder where the SAS data files are located in the LIBDAT statement. 

4. List all of the countries of interest in the parameter “COUNTRY” (by default, all TIMSS 2011 
countries are listed). 

5. Submit the edited code for processing by SAS. 

An example of the JOIN program is displayed in Exhibit 3.3. It joins the eighth grade student 
background data files (BSG) of all countries. All country data files are located in the folder titled 
“C:\TIMSS201 l\Data\SAS_Data” for the sake of this example. The resulting data file, BSGALLM5, 
also will be saved in this folder. 

Exhibit 3.3: Example of JOIN Program 


%LET TYPE = BSG ; 

LIBNAME LIBDAT "C:\TIMSS2011\Data\SAS_Data\" ; 

% MACRO DOIT ; 

%LET COUNTRY = < List of TIMSS 2011 countries > ; 

DATA &TYPE . ALLM5 ; 

SET %LET 1=1; 

%DO % WHILE ( % LENGTH ( %SCAN ( &COUNTRY , &I ) ) ) ; 

%LET CTRY = %SCAN (&COUNTRY, &I) ; 

LIBDAT . &TYPE&CTRY . M5 
%LET I = %EVAL ( &I +1) ; 

%END ; ; 

PROC SORT DATA = &TYPE . ALLM5 OUT = LIBDAT . &TYPE .ALLM5 ; 
BY &SORTVARS ; 

%MEND DOIT ; 


TIMSS 201 1 USER GUIDE FOR THE INTERNATIONAL DATABASE 

44 CHAPTER 3 



TIMSS 8r PIRLS 

International Study Center 

Lynch School of Education, Boston College 



3.6 SAS Macros to Compute Statistics and their Standard Errors 


This section describes the four SAS macros — JACKGEN, JACKPV, JACKREG, and JACKREGP — 
needed to compute specific statistics with their correct standard errors, along with sample SAS 
programs to demonstrate their use. Users are encouraged to modify the sample SAS programs and 
familiarize themselves with their functioning. The four SAS macros, however, do not require any 
modifications. 

Each SAS macro serves a specific analytical purpose. These macros ensure that analyses of the 
TIMSS 2011 data are done properly. Sampling weights are used and standard errors are computed 
using the jackknife repeated replication (JRR) method. Furthermore, achievement scores are based 
on sets of five plausible values that take into account the measurement error arising from the test 
design and the IRT scaling methodology. The macros that make use of these plausible values 
effectively perform five analyses — one for each plausible value — and aggregate the results to 
produce accurate estimates of achievement and standard errors that incorporate both sampling and 
imputation errors. 

The sample SAS programs presented in this section all use as input the SAS data file 
BSGALLM5, which contains the eighth grade student background data files of all participating 
countries. In all sample programs, <datpath> must be edited to specify the folder where the 
BSGALLM5 file is located, as well as <macpath> to indicate the folder where the SAS macros are 
located. 

Computing Means and their Standard Errors — JACKGEN 

The JACKGEN macro is used to compute percentages and means of continuous variables with their 
standard errors. We will demonstrate its use with a sample SAS program that calls the macro 
JACKGEN to compute the percentages of students within specified subgroups and their mean on a 
variable of choice. The macro also computes the appropriate standard errors for the percentages and 
means. However, this macro is not appropriate for analyzing achievement scores based on plausible 
values; the JACKPV macro should be used for this purpose. 

The JACKGEN macro is a self-contained program, located in the program file JACKGEN. SAS, 
and should not be modified. It essentially computes sets of replicate weights using the sampling and 
weighting variables, aggregates the data by subgroups using the replicate weights, and then computes 
and stores the desired statistics in a SAS working file called FINAL. 

The macro JACKGEN is included in a SAS program by issuing the following command: 


% INCLUDE "<macpath> JACKGEN. SAS" ; 


where <macpath> points to the folder in which the SAS macro JACKGEN. SAS is located. This 
macro requires that the following several parameters be specified as input when it is invoked: 

WGT The sampling weight to be used in the analysis. Generally, TOTWGT should be used. 

MATWGT should be used when analyzing mathematics teacher data, SCIWGT when 
analyzing science teacher data, and TCHWGT when analyzing all teacher data. 
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JKZ The variable that captures the assignment of cases (typically students) to sampling 

zones. The name of this variable in the TIMSS data files is JKZONE. 

JKR The variable that captures whether the case is to be dropped or have its weight 

doubled for each set of replicate weights. The name of this variable in the TIMSS data 
files is JKREP. 

NJKZ The number of replicate weights to be generated when computing the standard errors. 

The value of NJKZ should be set to 75, the maximum possible value across all 
participating countries. 

CVAR The list of variables that are to be used to define the subgroups. The list can consist of 
one or more variables. We recommend that users always include IDCNTRY as the 
first classification variable. 

DVAR The variable for which averages are to be computed. Only one variable can be listed 
and it should be a continuous variable. Plausible values of achievement scores should 
not be specified here. 

INFILE The name of the SAS data file that contains the data being analyzed. It is important to 
emphasize that this SAS data file must include only those cases that are of interest in 
the analysis. If users want to have specific cases excluded from the analysis 
(e.g., students with missing data), this should be done prior to invoking the macro. 

The JACKGEN macro is invoked by a SAS program using the conventional SAS notation for 
invoking macros. This involves listing the macro name followed by the list of parameters in 
parenthesis, each separated by a comma. For example, the JACKGEN macro invoked using the 
following statement: 


%JACKGEN ( TOTWGT , JKZONE, JKREP, 75, IDCNTRY ITSEX, BSDAGE, BSGALLM5) ; 


will compute the average age (BSDAGE) of eighth grade students by gender (ITSEX) and their 
standard errors within each country (IDCNTRY), using the weighting variable TOTWGT. It also 
will compute the percentages of girls and boys and their standard errors within each country. The 
data will be read from the data file BSGALLM5 and the standard errors will be computed based on 
75 sets of replicate weights. 

The results of the JACKGEN macro are stored in a SAS working file called FINAL, which is 
stored in the default folder used by SAS. The following variables are contained in this results file: 

Classification Variables 

All classification variables are kept in the results file. In the example invocation above, there are two 
classification variables: IDCNTRY and ITSEX. There is one record in the results file for each 
subgroup defined by the categories of the classification variables. 
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N 


This variable contains the number of valid cases for each subgroup defined by the classification 
variables. In the example, it is the number of girls and boys with valid data in each country’s sample. 

Weight Variable 

The weight variable contains the sum of weights within each subgroup defined by the classification 
variables. In the example, this variable is TOTWGT, because TOTWGT was specified as the 
weighting variable. This variable will be an estimate of the total population within each subgroup. 

MNX 

This variable contains the estimated averages of the specified analysis variable by subgroup. 

MNX_SE 

This variable contains the standard errors of the estimated averages by subgroup. 

PCT 

This variable contains the estimated percentages of students in each subgroup for the last 
classification variable listed. In the example, it is the percentage of girls and boys within each 
country. 

PCT_SE 

This variable contains the standard errors of the estimated percentages. 

The contents of the FINAL file can be printed using the SAS PRINT procedure. The sample SAS 
program that invokes the JACKGEN macro is presented in Exhibit 3.4, along with a printout of the 
results. This program is available in the International Database as the file called 
SAMPLEJACKGEN.SAS. It produces the average age of eighth grade girls and boys in all 
countries, although the exhibit shows the results only for the first four countries. 

As shown in the first two lines of the results in Exhibit 3.4, there were 3,746 girls in Australia’s 
eighth grade sample, representing 125,554 girls in the entire population. The average age of eighth 
grade girls in Australia was estimated to be 13.96, with a standard error of 0.01. Girls constituted 
49.83 percent of Australia’s eighth grade student population (standard error of 1.61). Conversely, 
Australia sampled 3,807 boys, representing 126,414 boys in the entire population. The estimated 
average age of eighth grade boys in Australia was 14.02, with a standard error of 0.01. Boys 
constituted 50.17 percent of Australia’s eighth grade student population (standard error of 1.61). 
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Exhibit 3.4: Sample SAS Program Invoking the SAS Macro JACKGEN with Results 


LIBNAME Til "<datpath>" ; 

% INCLUDE "<macpath> JACKGEN. SAS" ; 

DATA BSGALLM5 ; 

SET Til . BSGALLM5 ; 

WHERE NMISS (ITSEX, BSDAGE) = 0 ; 

PROC FORMAT LIBRARY = WORK ; 

VALUE COUNTRY 

< list TIMSS 2011 country formats > ; 

VALUE SEX 

1 = "GIRL" 

2 = "BOY" ; 

% JACKGEN ( TOTWGT , JKZONE, JKREP, 75, IDCNTRY ITSEX, BSDAGE, BSGALLM5) ; 

PROC PRINT DATA = FINAL NOOBS ; 

VAR IDCNTRY ITSEX N TOTWGT MNX MNX_SE PCT PCT_SE ; 

FORMAT IDCNTRY COUNTRY. ITSEX SEX. N 6.0 TOTWGT 10.0 
MNX MNX_SE PCT PCT_SE 6.2 ; 


IDCNTRY 

ITSEX 

N 

TOTWGT 

MNX 

MNX_SE 

PCT 

PCT_SE 

AUSTRALIA 

GIRLS 

3746 

125554 

13 . 96 

0 . 01 

49 . 83 

1.61 

AUSTRALIA 

BOYS 

3807 

126414 

14 . 02 

0 . 01 

50 . 17 

1.61 

BAHRAIN 

GIRLS 

2284 

6260 

14.39 

0 . 01 

49.66 

0 . 85 

BAHRAIN 

BOYS 

2351 

6346 

14.47 

0 . 01 

50.34 

0 . 85 

ARMENIA 

GIRLS 

2894 

17702 

14 . 60 

0 . 01 

48 . 93 

0 . 78 

ARMENIA 

BOYS 

2951 

18474 

14 . 54 

0 . 02 

51.07 

0 . 78 

CHILE 

GIRLS 

3133 

133160 

14 . 15 

0 . 01 

53 . 01 

1.46 

CHILE 

BOYS 

2702 

118020 

14.26 

0 . 02 

46.99 

1.46 


Computing Achievement Means and their Standard Errors — JACKPV 

The JACKPV macro computes percentages and average achievement scores using plausible values. 

It makes use of the sampling weights, the jackknifing algorithm to compute sampling variances, and 
the five plausible values to compute imputation variances. It effectively performs five analyses — one 
for each plausible value — and aggregates the results to produce accurate estimates of average 
achievement and standard errors that incorporate both sampling and imputation errors. 

A second sample program demonstrates the use of the JACKPV macro, which computes the 
percentages of students within specified subgroups and their average achievement scores. This SAS 
macro also computes the appropriate standard errors for those percentages and achievement 
averages. 

The JACKPV macro is a self-contained program, located in the program file JACKPV. SAS, and 
should not be modified. It essentially computes sets of replicate weights using the sampling and 
weighting variables, aggregates the data by subgroups using the replicate weights, and then computes 
and stores the desired statistics in a SAS working file called FINAL. The macro aggregates data 
across all plausible values to obtain the correct results. 

The SAS macro JACKPV is included in a SAS program by issuing the following command: 


% INCLUDE "<macpath> JACKPV. SAS" ; 
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where <macpath> points to the folder in which the SAS macro program JACKPV.SAS is located. 
This macro requires that the following several parameters be specified as input when it is invoked: 

WGT The sampling weight to be used in the analysis. Generally, TOTWGT should be used. 

MATWGT should be used when analyzing mathematics teacher data, SCIWGT when 
analyzing science teacher data, and TCHWGT when analyzing all teacher data. 

JKZ The variable that captures the assignment of cases to sampling zones. The name of 

this variable in the TIMSS data files is JKZONE. 

JKR The variable that captures whether the case is to be dropped or have its weight 

doubled for each set of replicate weights. The name of this variable in the TIMSS data 
files is JKREP. 

NJKZ The number of replicate weights to be generated when computing the standard errors. 

The value of NJKZ should be set to 75, the maximum possible value across all 
participating countries. 

CVAR The list of variables that are to be used to define the subgroups. The list can consist of 

one or more variables. We recommend that users always include IDCNTRY as the 
first classification variable. 

ROOTPV The variable root used to identify the set of plausible values for the achievement score 
of interest. It corresponds to the first seven characters of the plausible values variable 
name. For example, the root of the eighth grade overall mathematics plausible values 
is BSMMATO, the root of the eighth grade overall science plausible values is 
BSSSCIO. 

NPV The number of plausible values that will be used for the analysis. Generally, it is set to 

five in order to use all five available plausible values for analysis. 

INFILE The name of the SAS data file that contains the data being analyzed. It is important to 
emphasize that this SAS data file must include only those cases that are of interest in 
the analysis. If users want to have specific cases excluded from the analysis 
(e.g., students with missing data), this should be done prior to invoking the macro. 

The JACKPV macro is invoked by a SAS program using the conventional SAS notation for 
invoking macros. This involves listing the macro name followed by the list of parameters in 
parenthesis, each separated by a comma. For example, the JACKPV macro invoked using the 
following statement: 


%JACKPV (TOTWGT, JKZONE, JKREP, 75, IDCNTRY ITSEX, BSMMATO, 5, BSGALLM5 ) ; 


will compute the average eighth grade mathematics achievement (BSMMATO 1 through 
BSMMAT05) by gender (ITSEX) within each country (IDCNTRY) and their standard errors, using 
the weighting variable TOTWGT. The macro uses all five plausible values to compute these 
statistics. It also will compute the percentages of girls and boys within each country, along with their 
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standard errors. The data will be read from the data file BSGALLM5 and the standard errors will be 
computed based on 75 sets of replicate weights. 

The results of the JACKPV macro are stored in a SAS working file called FINAL, which is 
stored in the default folder used by SAS. The following variables are contained in this results file: 

Classification Variables 

All classification variables are kept in the results file. In this example, there are two classification 
variables: IDCNTRY and ITSEX. There is one record in the results file for each subgroup defined by 
the categories of the classification variables. 

N 

This variable contains the number of valid cases for each subgroup defined by the classification 
variables. In the example, it is the number of girls and boys with valid data in each country’s sample. 

Weight Variable 

The weight variable contains the sum of weights within each subgroup defined by the classification 
variables. In the example, this variable is called TOTWGT, because TOTWGT was specified as the 
weighting variable. This variable will be an estimate of the total population within each subgroup. 

MNPV 

This variable contains the estimated average achievement by subgroup, based on the plausible 
values. 

MNPV_SE 

This variable contains the standard errors of the estimated average achievement by subgroup, based 
on the plausible values. 

PCT 

This variable contains the estimated percentages of students in each subgroup for the last 
classification variable listed. In the example, it is the percentage of girls and boys within each 
country. 

PCT_SE 

This variable contains the standard errors of the estimated percentages. 

The contents of the FINAL file can be printed using the SAS PRINT procedure. The sample SAS 
program that invokes the JACKPV macro is shown in Exhibit 3.5, along with a printout of the 
results. This program is available in the International Database as the file called 
SAMPLEJACKPV.SAS. It produces the average eighth grade mathematics achievement for girls and 
boys in all countries, although Exhibit 3.5 gives the results only for the first four countries. 
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Exhibit 3.5: Sample SAS Program Invoking the SAS Macro JACKPV with Results 


LIBNAME Til "<datpath>" ; 

% INCLUDE "<macpath> JACKPV. SAS" ; 

DATA BSGALLM5 ; 

SET Til . BSGALLM5 ; 

WHERE NMISS (ITSEX) = 0 ; 

PROC FORMAT LIBRARY = WORK ; 

VALUE COUNTRY 

< list TIMSS 2011 country formats > ; 

VALUE SEX 

1 = "GIRL" 

2 = "BOY" ; 

%JACKPV ( TOTWGT , JKZONE, JKREP, 75, IDCNTRY ITSEX, BSMMATO , 5, BSGALLM5 ) ; 

PROC PRINT DATA = FINAL NOOBS ; 

VAR IDCNTRY ITSEX N TOTWGT MNPV MNPV_SE PCT PCT_SE ; 

FORMAT IDCNTRY COUNTRY. ITSEX SEX. N 6.0 TOTWGT 10.0 
MNPV MNPV_SE PCT PCT_SE 6.2 ; 


IDCNTRY 

ITSEX 

N 

TOTWGT 

MNPV 

MNPV_SE 

PCT 

PCT_SE 

AUSTRALIA 

GIRLS 

3747 

125555 

500.41 

4 . 72 

49 . 83 

1.61 

AUSTRALIA 

BOYS 

3809 

126429 

509 . 16 

7.26 

50 . 17 

1.61 

BAHRAIN 

GIRLS 

2288 

6276 

430 . 78 

2.51 

49.72 

0 . 85 

BAHRAIN 

BOYS 

2352 

6348 

387 .89 

3 . 07 

50.28 

0 . 85 

ARMENIA 

GIRLS 

2894 

17702 

471.52 

3 . 07 

48 . 93 

0 . 78 

ARMENIA 

BOYS 

2952 

18479 

461.86 

3.21 

51 . 07 

0 . 78 

CHILE 

GIRLS 

3133 

133160 

409.46 

3.23 

53 . 01 

1.46 

CHILE 

BOYS 

2702 

118020 

423 . 94 

3 . 05 

46 . 99 

1.46 


As shown in the first two lines of the results in Exhibit 3.5, the average mathematics achievement 
of eighth grade girls in Australia was estimated at 500.41, with a standard error of 4.72. The average 
mathematics achievement of eighth grade boys in Australia was estimated at 509. 16, with a standard 
error of 7.26. 


Computing Regression Coefficients and their Standard Errors — JACKREG 

The JACKREG macro performs a multiple linear regression between a dependent variable and a set 
of independent variables. A third sample program demonstrates the use of the JACKREG macro, 
which computes the regression coefficients and their standard errors. This macro is not appropriate 
for regression analyses using achievement scores as the dependent variable. The JACKREGP macro 
should be used for this purpose. 

The JACKREG macro is a self-contained program, located in the program file JACKREG. SAS, 
and should not be modified. It computes sets of replicate weights using the sampling and weighting 
variables, performs a linear regression by subgroup using the replicate weights, and then computes 
and stores the desired statistics in a SAS working file called REG. 

The SAS macro JACKREG is included in a SAS program by issuing the following command: 


% INCLUDE "<macpath> JACKREG. SAS" ; 
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where <macpath> points to the specific folder in which the SAS macro program JACKREG. SAS is 
located. This macro requires that the following several parameters be specified as input when it is 
invoked: 

WGT The sampling weight to be used in the analysis. Generally, TOTWGT should be used. 

MATWGT should be used when analyzing mathematics teacher data, SCIWGT when 
analyzing science teacher data, and TCHWGT when analyzing all teacher data. 

JKZ The variable that captures the assignment of cases to sampling zones. The name of 

this variable in the TIMSS data files is JKZONE. 

JKR The variable that captures whether the case is to be dropped or have its weight 

doubled for each set of replicate weights. The name of this variable in the TIMSS data 
files is JKREP. 

NJKZ The number of replicate weights to be generated when computing the standard errors. 
The value of NJKZ should be set to 75, the maximum possible value across all 
participating countries. 

CVAR The list of variables that are to be used to define the subgroups. The list can consist of 
one or more variables. We recommend that users always include IDCNTRY as the 
first classification variable. 

XVAR The list of independent variables used as predictors in the regression model. The 
independent variables can be either continuous or categorical (e.g., ITSEX). 

DVAR The dependent variable to be predicted by the list of independent variables specified 
in XVAR. Only one variable can be listed, and plausible values of achievement scores 
should not be specified here. 

INFILE The name of the SAS data file that contains the data being analyzed. It is important to 
emphasize that this SAS data file must include only those cases that are of interest in 
the analysis. If users want to have specific cases excluded from the analysis, for 
example students with missing data, this should be done prior to invoking the macro. 

The JACKREG macro is invoked by a SAS program using the conventional SAS notation for 
invoking macros. This involves listing the macro name followed by the list of parameters in 
parenthesis, each separated by a comma. For example, the JACKREG macro invoked using the 
following statement: 


%JACKREG (TOTWGT, JKZONE, JKREP, 75, IDCNTRY, REGSEX, BSBGSLM, BSGALLM5 ) ; 


will perfonn a linear regression with gender (REGSEX) as a predictor of the eighth grade students’ 
score on the Students Like Learning Mathematics contextual scale (BSBGSLM), using the weighting 
variable TOTWGT. “ It will compute the regression coefficients and their standard errors. The data 


2 The Students Like Learning Mathematics contextual scale is described in the section Student-level Analysis with a Contextual Scale. 
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will be read from the data file BSGALLM5 and the standard errors will be computed based on 75 
replicate weights. 

The results of the JACKREG macro are stored in a SAS working file called REG, which is stored 
in the default folder used by SAS. The following variables are contained in this results file: 

Classification Variables 

All classification variables are kept in the results file. In this example, there is a single classification 
variable IDCNTRY. There is one record in the results file for each subgroup defined by the 
categories of the classification variables. 

N 

This variable contains the number of valid cases for each subgroup defined by the classification 
variables. In the example, it is the number of students with valid data in each country’s sample. 

MULT_RSQ 

The squared multiple correlation coefficient (R ) for the regression model applied in each subgroup. 
SS_RES, SS_REG, SS_TOTAL 

The residual, regression, and total weighted sums of squares for the regression model applied in each 
subgroup. 

Regression Coefficients and Standard Errors (B## and B##.SE) 

The regression coefficients for the intercept and the predictor variables with their respective standard 
errors. The regression coefficients are numbered sequentially, starting with zero (BOO) for the 
intercept, and based on the order of the predictor variables as specified in the parameter XV AR. 

The contents of the REG file can be printed using the SAS PRINT procedure. The sample SAS 
program that invokes the JACKREG macro is displayed in Exhibit 3.6, along with a printout of the 
results. This program is available in the International Database as the file called 
SAMPLE JACKREG. SAS. It performs a linear regression in each country, with the variable 
REGSEX as a predictor of the eighth grade students’ score on the Students Like Learning 
Mathematics contextual scale (BSBGSLM). The exhibit displays the results for the first four 
countries. 

The regression performed by the sample program uses the independent variable REGSEX, which 
is a “dummy-coded” version of ITSEX, such that the value “0” represents the girls and the value “1” 
represents the boys. By performing this recoding, the intercept BOO will be the estimated average 
score of eighth grade girls, whereas the regression coefficient B01 will be the estimated increase in 
average score for boys. This will allow us to determine if the difference in average score between 
girls and boys is statistically significant. 

As shown in the first line of the results in Exhibit 3.6, the estimated average score of eighth grade 
girls in Australia on the BSBGSLM scale (BOO) was 9.10, with a standard error of 0.07. The eighth 
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grade boys in Australia were an estimated 0.28 points higher (B01) than the girls on that scale, with 
an estimated standard error of 0.08. Note that this difference is statistically significant at the 95% 
confidence level. 

Exhibit 3.6: Sample SAS Program Invoking the SAS Macro JACKREG with Results 


LIBNAME Til "<datpath>" ; 

% INCLUDE "<macpath> JACKREG. SAS" ; 

DATA BSGALLM5 ; 

SET Til . BSGALLM5 ; 

WHERE NMISS (ITSEX, BSBGSLM) = 0 ; 

SELECT (ITSEX) ; 

WHEN (1) REGSEX = 0 ; * GIRLS ; 

WHEN (2) REGSEX = 1 ; * BOYS ; 

OTHERWISE REGSEX = . ; 

END ; 

PROC FORMAT LIBRARY = WORK ; 

VALUE COUNTRY 

< list TIMSS 2011 country formats > ; 

% JACKREG ( TOTWGT , JKZONE, JKREP, 75, IDCNTRY, REGSEX, BSBGSLM, BSGALLM5 ) ; 
PROC PRINT DATA = REG NOOBS ; 

VAR IDCNTRY N MULT_RSQ SS_TOTAL SS_REG BOO B00_SE B01 B01_SE ; 

FORMAT IDCNTRY COUNTRY. N 6.0 MULT_RSQ 5.3 SS_TOTAL SS_REG 10.0 
BOO B00_SE B01 B01_SE 6.2 ; 


IDCNTRY 

N 

MULT_RSQ 

SS_TOTAL 

SS_REG 

BOO 

B00_SE 

B01 

B01_SE 

AUSTRALIA 

7389 

0 . 005 

910209 

4882 

9.18 

0 . 07 

0.28 

0 . 08 

BAHRAIN 

4581 

0 . 000 

52295 

6 

9.75 

0 . 05 

0 . 05 

0 . 07 

ARMENIA 

5626 

0 . 001 

128047 

162 

10 . 94 

0 . 06 

-0 . 14 

0 . 06 

CHILE 

5772 

0 . 012 

949606 

11772 

9.56 

0 . 05 

0.44 

0 . 06 


Computing Regression Coefficients and their Standard Errors with Achievement — JACKREGP 

The JACKREGP macro is used to perform a multiple linear regression between a set of plausible 
values as the dependent variable and a set of independent variables. It computes the regression 
coefficients and their standard errors, making use of the sampling weights, the jackknifing algorithm 
to compute sampling variances, and the five plausible values to compute imputation variances. It 
effectively performs five regression analyses — one for each plausible value — and aggregates the 
results to produce accurate estimates of the regression coefficients and standard errors that 
incorporate both sampling and imputation errors. Below, we present a fourth sample program to 
demonstrate the use of the JACKREGP macro. 

The JACKREGP macro is a self-contained program, located in the program file 
JACKREGP. SAS, and should not be modified. It computes sets of replicate weights using the 
sampling and weighting variables, performs a multiple linear regression by subgroups using the 
replicate weights, and then computes and stores the desired statistics in a SAS working file called 
REG. 

The SAS macro JACKREGP is included in a SAS program by issuing the following command: 


% INCLUDE "<raacpath> JACKREGP. SAS" ; 
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where <macpath> points to the specific folder in which the SAS macro program JACKREGP.SAS is 
located. This macro requires that the following several parameters be specified as input when it is 
invoked: 

WGT The sampling weight to be used in the analysis. Generally, TOTWGT should be used. 

MATWGT should be used when analyzing mathematics teacher data, SCIWGT when 
analyzing science teacher data, and TCHWGT when analyzing all teacher data. 

JKZ The variable that captures the assignment of cases to sampling zones. The name of 

this variable in the TIMSS data files is JKZONE. 

JKR The variable that captures whether the case is to be dropped or have its weight 

doubled for each set of replicate weights. The name of this variable in the TIMSS data 
files is JKREP. 

NJKZ The number of replicate weights to be generated when computing the standard errors. 
The value of NJKZ should be set to 75, the maximum possible value across all 
participating countries. 

CVAR The list of variables that are to be used to define the subgroups. The list can consist of 
one or more variables. We recommend that users always include IDCNTRY as the 
first classification variable. 

XVAR The list of independent variables used as predictors in the regression model. The 
independent variables can be either continuous or categorical, such as ITSEX for 
example. 

ROOTPV The variable root used to identify the set of plausible values for the achievement score 
of interest. It corresponds to the first seven characters of the plausible values variable 
name. For example, the root of the eighth grade overall mathematics plausible values 
is BSMMATO, the root of the eighth grade overall science plausible values is 
BSSSCIO. 

NPV The number of plausible values that will be used for the analysis. Generally, it is set to 
five to use all five available plausible values for analysis. 

INFILE The name of the SAS data file that contains the data being analyzed, this number is 
important to emphasize that this SAS data file must include only those cases that are 
of interest in the analysis. If users want to have specific cases excluded from the 
analysis, for example students with missing data, this should be done prior to 
invoking the macro. 

The JACKREGP macro is invoked by a SAS program using the conventional SAS notation for 
invoking macros. This involves listing the macro name followed by the list of parameters in 
parenthesis, each separated by a comma. For example, the JACKREGP macro invoked using the 
following statement: 


% JACKREGP (TOTWGT, JKZONE, JKREP, 75, IDCNTRY, REGSEX, BSMMATO, 5, BSGALLM5 ) ; 
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will perform a linear regression with gender (REGSEX) as a predictor of eighth grade mathematics 
achievement based on its five plausible values (BSMMAT01 through BSMMAT05), using the 
weighting variable TOTWGT. It will compute the regression coefficients and their standard errors. 
The data will be read from the data file BSGALLM5 and the standard errors will be computed based 
on 75 replicate weights. 

The results of the JACKREGP macro are stored in a SAS working file called REG, which is 
stored in the default folder used by SAS. The following variables are contained in this results file: 

Classification Variables 

All classification variables are kept in the results file. In this example, there is a single classification 
variable IDCNTRY. There is one record in the results file for each subgroup defined by the 
categories of the classification variables. 

N 

This variable contains the number of valid cases for each subgroup defined by the classification 
variables. In the example, it is the number of students with valid data in each country’s sample. 

MULT_RSQ 

The squared multiple correlation coefficient (R ) for the regression model applied in each subgroup. 
SS_RES, SS_REG, SS_TOTAL 

The residual, regression, and total weighted sums of squares for the regression model applied in each 
subgroup. 

Regression Coefficients and Standard Errors (B## and B##.SE) 

The regression coefficients for the predictor variables and the intercept with their respective standard 
errors. The regression coefficients are numbered sequentially, starting with zero (BOO) for the 
intercept, and based on the order of the predictor variables as specified in the parameter XVAR. 

The contents of the REG file can be printed using the SAS PRINT procedure. The sample SAS 
program invoking the JACKREGP macro is presented in Exhibit 3.7, along with a printout of the 
results. This program is available in the International Database as the file called 
SAMPLE JACKREGP. SAS. It performs a linear regression in each country, with the variable 
REGSEX as a predictor of eighth grade mathematics achievement. The exhibit displays the results 
for the first four countries. 

The regression performed by our sample program uses the variable REGSEX that was defined in 
our previous example. By using REGSEX, the intercept BOO will be the estimated average 
mathematics achievement of eighth grade girls, whereas the regression coefficient B01 will be the 
estimated difference in the average mathematics achievement of boys. This will allow us to 
detennine if eighth grade mathematics achievement is significantly different between girls and boys. 
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Exhibit 3.7: Sample SAS Program Invoking the SAS Macro JACKREGP with Results 


LIBNAME Til "<datpath>" ; 

% INCLUDE " <macpath> JACKREGP . SAS " ; 

DATA BSGALLM5 ; 

SET Til . BSGALLM5 ; 

WHERE NMISS (ITSEX) = 0 ; 

SELECT (ITSEX) ; 

WHEN (1) REGSEX = 0 ; * GIRLS ; 

WHEN (2) REGSEX = 1 ; * BOYS ; 

OTHERWISE REGSEX = . ; 

END ; 

PROC FORMAT LIBRARY = WORK ; 

VALUE COUNTRY 

< list TIMSS 2011 country formats > ; 

%JACKREGP (TOTWGT, JKZONE, JKREP, 75, IDCNTRY, REGSEX, BSMMATO , 5, BSGALLM5 ) ; 
PROC PRINT DATA = REG NOOBS ; 

VAR IDCNTRY N MULT_RSQ SS_TOTAL SS_REG BOO B00_SE B01 B01_SE ; 

FORMAT IDCNTRY COUNTRY. N 6.0 MULT_RSQ 5.3 SS_TOTAL SS_REG 10.0 
BOO B00_SE B01 B01_SE 6.2 ; 


IDCNTRY 

N 

MULT_RSQ 

SS_TOTAL 

SS_REG 

BOO 

B00_SE 

B01 

B01_SE 

AUSTRALIA 

7556 

0 .003 

1838740030 

4904428 

500.41 

4 . 72 

8.75 

6 . 90 

BAHRAIN 

4640 

0 . 046 

125166261 

5807054 

430.78 

2.51 

-42 . 89 

3 . 99 

ARMENIA 

5846 

0 . 003 

297563160 

853896 

471.52 

3 . 07 

-9.66 

3 . 10 

CHILE 

5835 

0 .008 

1593853260 

13134187 

409.46 

3.23 

14.48 

3.63 


From the first line of the results shown in Exhibit 3.7, the estimated average mathematics 
achievement of eighth grade girls in Australia (BOO) was 500.41, with a standard error of 4.72. Note 
that these are the same results obtained from the JACKPV sample program (Exhibit 3.5). The eighth 
grade boys had an estimated average mathematics achievement 8.75 points (B01) higher than girls, 
with an estimated standard error of 6.90. Note that this difference is not statistically significant at the 
95% confidence level. 

3.7 TIMSS Analyses with Student-level Variables 

Many analyses of the TIMSS 2011 data can be undertaken using only student-level data. Examples 
in the previous section illustrated the functioning of the SAS macros. This section presents examples 
of actual analyses reported in TIMSS 2011 International Results in Mathematics (Mullis, Martin, 

Foy, & Arora, 2012), using SAS programs provided in the TIMSS 2011 International Database. 

A first example computes national average achievement, whereas a second example computes 
national average achievement by gender. In both cases, the SAS macros use the sampling weights, 
implement the jackknife repeated replication method to compute appropriate sampling errors, 
effectively perform the computations five times (once for each plausible value), and aggregate the 
results to produce accurate estimates of average achievement and standard errors that incorporate 
both sampling and imputation errors. A third example expands on the second example by performing 
a test of significance on the gender difference using regression. Finally, a fourth example computes 
the average scale score for one of the newly developed contextual scales, along with the percentages 
of students, with their average achievement, for the categories of the scale’s corresponding index. 
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In general, to perform student-level analyses using the student background data fdes, users 
should do the following: 

1 . Identify the variables of interest in the student background data files and note any specific 
national adaptations to the variables. 

2. Retrieve the relevant variables from the student background data files, including the plausible 
values of achievement if required, classification variables, identification variables, sampling and 
weighting variables, and any other variables used in the selection of cases. 

3. Perform any necessary variable transformations or recodes. 

4. Use the macros JACKGEN and JACKREG, or JACKPV and JACKREGP if plausible values are 
involved, with the appropriate parameters. 

5. Specify the location of the data files (<datpath>) and the macros (<macpath>). 

6. Print the results file. 

Student-level Analysis with Achievement 

In our first example, we want to replicate the analysis of the overall distribution of mathematics 
achievement. These results are presented in Exhibit 1.2 of TIMSS 2011 International Results in 
Mathematics and are repeated here in Exhibit 3.8. Because the results in this exhibit are based on 
plausible values, we need to make sure that we include them when we create the file using the merge 
module, and also to indicate that our analysis will make use of achievement scores. 

We will find all of the variables we need for this analysis in the student background data files, 
including the five plausible values of eighth grade mathematics achievement (BSMMAT01 through 
BSMMAT05), the student sampling weight (TOTWGT), the variables that contain the jackknife 
replication information (JKZONE and JKREP), and the variable containing the country identification 
code (IDCNTRY). In this analysis, we will use the data for all available countries. We used the JOIN 
program, described earlier in this chapter, to join the student background data files for all countries 
into a single file called BSGALLM5. 

The SAS program used to perform this first example is presented in Exhibit 3.9 and is included in 
the International Database under the name EXAMPLE1.SAS. The results obtained from this program 
are displayed in Exhibit 3.10, although only the results of first four countries are shown for the sake 
of brevity. 

In Exhibit 3.10, the results are displayed one line per country, with the countries identified in the 
first column. The second column reports the number of valid cases, and the third the sum of weights 
of the sampled students. The next four columns report the estimated average mathematics 
achievement and its standard error, followed by the percentage of students in each category and its 
standard error. As shown in the first line, the average mathematics achievement of eighth grade 
students in Australia was 504.80, with a standard error of 5.09. In this example, only IDCNTRY was 
used as a classification variable; thus, the estimated percentages are of little value because they 
represent the proportion of the TOTWGT column for each country. 
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Exhibit 3.8: Exhibit of Example Student-level Analysis with Achievement, Taken from 

TIMSS 201 7 International Results in Mathematics (Exhibit 1.2) 


Exhibit 1 .2: Distribution of Mathematics Achievement 


Country 

Korea, Rep. of 
2 Singapore 
Chinese Taipei 
Hong Kong SAR 
Japan 

2 Russian Federation 

3 Israel 
Finland 

2 United States 


Norway 

Armenia 

Romania 

United Arab Emirates 
Turkey 
Lebanon 
Malaysia 
1 Georgia 
Thailand 

^ Macedonia, Rep. of 
Tunisia 
Chile 

^ Iran, Islamic Rep. of 


V Qatar 


Average 
Scale Score 
613 (2.9) O 


611 (3.8) 


609 (3.2) 


586 (3.8) 


570 (2.6) 


539 (3.6) 


516 (4.1) 


514 (2.5) 


509 (2.6) 


* England 

507 (5.5) 


Hungary 

505 (3.5) 


Australia 

505 (5.1) 


Slovenia 

505 (2.2) 

O 

1 Lithuania 

502 (2.5) 


TIMSS Scale Centerpoint 

500 


Italy 

498 (2.4) 


New Zealand 

488 (5.5) 

s 

Kazakhstan 

487 (4.0) 

® 

Sweden 

484 (1.9) 

® 

Ukraine 

479 (3.9) 

® 


475 (2.4) ® 

467 (2.7) ® 


458 (4.0) ® 


456 (2.1) 
452 (3.9) 
449 (3.7) 
440 (5.4) 
431 (3.8) 
427 (4.3) 
426 (5.2) 
425 (2.8) 
416 (2.6) 
415 (4.3) 


® 


TIMSS 2011 
Mathematics 


Mathematics Achievement Distribution 


410 (3.1) ® 


Y Bahrain 
^ Jordan 

v Palestinian Nat'l Auth. 
v Saudi Arabia 

Y Indonesia 

^ Syrian Arab Republic 
* Morocco 
v Oman 
>k Ghana 


409 (2.0) ® 


406 (3.7) 
404 (3.5) 


® 


394 (4.6) ® 


386 (4.3) ® 


380 (4.5) ® 


371 (2.0) ® 


366 (2.8) ® 


331 (4.3) ® 


Country 

Average 
Scale Score 

Mathematics Achievement Distribution 

Ninth Grade Participants 

^ Botswana 

397 (2.5) ® 

■ ■ ■ ■ 

* South Africa 

352 (2.5) ® 


2 * Honduras 

338 (3.7) ® 



Benchmarking Participants 


1 2 Massachusetts, US 
1 Minnesota, US 
1 3 North Carolina, US 
Quebec, Canada 
1 2 Indiana, US 

1 Colorado, US 

1 2 Connecticut, US 
1 2 Florida, US 

2 Ontario, Canada 
2 Alberta, Canada 


561 (5.3) 


545 (4.6) O 

537 (6.8) O 

532 (2.3) O 


522 (5.1) 


518 (4.9) O 

518 (4.8) O 

513 (6.4) O 

512 (2.5) O 


505 (2.6) 


1 2 California, US 

493 (4.9) 


Dubai, UAE 

478 (2.1) 

® 

1 Alabama, US 

466 (5.9) 

® 

Abu Dhabi, UAE 

449 (3.7) 

® 


I” 

100 


O Country average significantly higher than 
the centerpoint of the TIMSS 8th grade scale 
® Country average significantly lower than 
the centerpoint of the TIMSS 8th grade scale 


Percentiles of Performance 



95% Confidence Interval for Average (±2SE) 


/■K Average achievement not reliably measured because the percentage of students with achievement too low for estimation exceeds 25%. 

V Reservations about reliability of average achievement because the percentage of students with achievement too low for estimation does not exceed 25% but exceeds 1 5%. 
See Appendix C.3 for target population coverage notes 1, 2, and 3. See Appendix C.9 for sampling guidelines and sampling participation notes t, *, and t. 

( ) Standard errors appear In parentheses. Because of rounding some results may appear Inconsistent. 
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Exhibit 3.9: Example SAS Program for Student-level Analysis with Achievement (EXAMPLE1 .SAS) 


LIBNAME Til "<datpath>" ; 

% INCLUDE " <macpath> JACKPV . SAS " ; 

DATA BSGALLM5 ; 

SET Til . BSGALLM5 ; 

PROC FORMAT LIBRARY = WORK ; 

VALUE COUNTRY 

< list TIMSS 2011 country formats > ; 

%JACKPV ( TOTWGT , JKZONE, JKREP, 75, IDCNTRY ITSEX, BSMMATO , 5, BSGALLM5 ) ; 

PROC PRINT DATA = FINAL NOOBS ; 

VAR IDCNTRY N TOTWGT MNPV MNPV_SE PCT PCT_SE ; 

FORMAT IDCNTRY COUNTRY. N 6.0 TOTWGT 10.0 
MNPV MNPV_SE PCT PCT_SE 6.2 ; 


Exhibit 3.1 0: Output for Example Student-level Analysis with Achievement (EXAMPLE 1 ) 


IDCNTRY 

N 

TOTWGT 

MNPV 

MNPV_SE 

PCT 

PCT_SE 

AUSTRALIA 

7556 

251985 

504 . 80 

5 . 09 

1 . 18 

0 . 02 

BAHRAIN 

4640 

12625 

409.22 

1 . 96 

0 . 06 

0 . 00 

ARMENIA 

5846 

36181 

466 . 59 

2 . 73 

0 . 17 

0 . 00 

CHILE 

5835 

251180 

416.27 

2 . 59 

1.18 

0 . 03 


Student-level Analysis with Achievement by Gender 

In our second example, we want to replicate another set of results presented in TIMSS 2011 
International Results in Mathematics . We are interested in investigating the relationship between 
eighth grade students’ gender and mathematics achievement. These results are presented in 
Exhibit 1.11 of TIMSS 2011 International Results in Mathematics and repeated here in Exhibit 3.11. 
Because the results in this exhibit are based on plausible values, we must make sure they are 
included when creating the input file, and also indicate that this analysis will make use of 
achievement scores. 

After reviewing the appropriate codebook, the variable ITSEX in the student background data 
files contains information on the gender of students. We then proceed to read from the student 
background data files our variable of interest (ITSEX), the five plausible values of eighth grade 
mathematics achievement (BSMMATO 1 through BSMMAT05), the student sampling weight 
(TOTWGT), the variables that contain the jackknifing information (JKZONE and JKREP), and the 
country identification variable (IDCNTRY). Again, we will use the data of all available countries 
contained in the file BSGALLM5. 

The SAS program that implements this second example is presented in Exhibit 3.12 and is 
included in the International Database under the name EXAMPLE2.SAS. Note that one of the steps 
in this program is to select only those students who have non-missing data in our variable of interest 
ITSEX. 
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Exhibit 3.1 1 : Exhibit of Example Student-level Analysis with Achievement by Gender, Taken from 
TIMSS20 7 7 International Results in Mathematics (Exhibit 1.11) 


Exhibit 1.11: Average Mathematics Achievement by Gender 


TIMSS 2011 
Mathematics 




Girls 

Boys 

Difference 

Gender Difference 

Country 

Percent of 
Students 

Average Scale 
Score 

Percent of 
Students 

Average Scale 
Score 

(Absolute Girls Boys 

Value) Scored Higher Scored Higher 

* Morocco 

47 (0.8) 

371 (2.3) 

53 (0.8) 

371 (2.7) 

0(3.2) 




2 Russian Federation 

49 (0.9) 

539 (3.8) 

51 (0.9) 

539 (3.9) 

1 (2.9) 


i 


Kazakhstan 

49 (0.8) 

486 (4.1) 

51 (0.8) 

488 (4.5) 

2 (3.3) 


i 


Norway 

49 (0.7) 

476 (2.9) 

51 (0.7) 

473 (2.9) 

3 (3.1) 


« 


* England 

48 (2.0) 

508 (5.7) 

52 (2.0) 

505 (6.6) 

3 (5.6) 


• 


1 Georgia 

47 (0.9) 

430 (4.1) 

53 (0.9) 

432 (4.4) 

3 (4.0) 


■ 


Ukraine 

50 (1.0) 

478 (4.0) 

50 (1.0) 

481 (4.9) 

3 (4.4) 


■ 


2 United States 

51 (0.6) 

508 (2.9) 

49(0.6) 

511 (2.8) 

4(2.2) 


■ 


Sweden 

48 (0.9) 

486 (2.1) 

52 (0.9) 

482 (2.4) 

4(2.4) 


■ 


Finland 

48(1.1) 

516 (2.7) 

52(1.1) 

512 (2.7) 

4(2.3) 


■ 


Slovenia 

49 (0.9) 

502 (2.4) 

51 (0.9) 

507 (2.8) 

5 (2.8) 


■ 


Hungary 

49(1.1) 

502 (3.9) 

51 (1.1) 

508 (3.9) 

6 (3.5) 


■ 


Flong Kong SAR 

49 (1.6) 

588 (5.0) 

51 (1.6) 

583 (4.3) 

6(5.5) 


■ 


Chinese Taipei 

48 (1.0) 

613 (3.7) 

52 (1.0) 

606 (3.8) 

6(4.1) 


■ 


Korea, Rep. of 

52 (2.5) 

610 (3.5) 

48(2.5) 

616(3.1) 

6(3.1) 


■ 


^ Iran, Islamic Rep. of 

46 (2.3) 

411 (5.9) 

54(2.3) 

418 (5.9) 

7 (8.1) 


■ 


^ Macedonia, Rep. of 

49 (0.9) 

430 (5.8) 

51 (0.9) 

423 (5.6) 

7 (4.7) 


■ 


Japan 

49(1.1) 

566 (3.1) 

51 (1.1) 

574 (3.5) 

8 (4.1) 


■ 


3 Israel 

50 (1.6) 

520 (3.9) 

50(1.6) 

512 (5.2) 

8(4.4) 


■ 


2 Singapore 

49 (0.7) 

615 (3.7) 

51 (0.7) 

607 (4.5) 

9(3.5) 


■ 


Turkey 

49 (0.7) 

457 (3.8) 

51 (0.7) 

448 (4.7) 

9(3.5) 


■ 


Australia 

50 (1.6) 

500 (4.7) 

50(1.6) 

509 (7.3) 

9(6.9) 


■ 


1 Lithuania 

49 (0.7) 

507 (2.6) 

51 (0.7) 

498 (3.2) 

9(3.0) 


■ 


Armenia 

49 (0.8) 

472 (3.1) 

51 (0.8) 

462 (3.2) 

10(3.1) 




^ Syrian Arab Republic 

50 (1.7) 

375 (5.3) 

50(1.7) 

385 (5.3) 

11 (5.7) 




Italy 

49 (0.9) 

493 (2.9) 

51 (0.9) 

504 (2.8) 

11 (2.9) 




Romania 

48 (0.9) 

464 (4.6) 

52 (0.9) 

453 (4.2) 

11 (3.6) 


■ 


v Qatar 

50 (3.3) 

415 (5.8) 

50(3.3) 

404 (5.5) 

11 (9.5) 




Lebanon 

55 (1.9) 

444 (4.2) 

45 (1.9) 

456 (4.7) 

12 (4.7) 


■ 


v Indonesia 

50 (1.2) 

392 (4.9) 

50 (1.2) 

379 (4.5) 

13 (4.0) 




Chile 

53 (1.5) 

409 (3.2) 

47 (1 j) 

424 (3.0) 

14 (3.6) 




f Saudi Arabia 

48(1.2) 

401 (4.1) 

52 (12) 

387 (8.0) 

15 (8.9) 


■■ 


Tunisia 

52 (0.7) 

417(3.1) 

48 (0.7) 

433 (3.1) 

17(2.5) 




United Arab Emirates 

50(1.7) 

464 (2.7) 

50(1.7) 

447 (3.1) 

17 (4.2) 





Thailand 

55 (1.6) 

435 (4.2) 

45 (1.6) 

417 (5.3) 

18 (4.4) 


■■■ 


New Zealand 

47 (2.0) 

478 (5.5) 

53 (2.0) 

496 (6.2) 

18 (4.7) 




Malaysia 

51 (12) 

449 (5.2) 

49(1.2) 

430 (6.2) 

19 (4.4) 




'f Palestinian Nat'l Auth. 

52 (1.7) 

415 (4.2) 

48(1.7) 

392 (5.6) 

23 (7.0) 



>k Ghana 

47 (0.8) 

318 (4.8) 

53 (0.8) 

342 (4.3) 

23 (2.9) 



y Jordan 

49(1.7) 

420 (4.3) 

51 (1.7) 

392 (5.9) 

28 (7.4) 



v Bahrain 

50 (0.8) 

431 (2.5) 

50 (0.8) 

388 (3.1) 

43 (4.0) l 



*p Oman 

51 (2.1) 

397 (3.1) 

49 (2.1) 

334 (3.8) 

63 (4.6) 



[ International Avg. 

50 (02) 

| 469 (0.6) 

50 (0.2) | 

465 (0.7) E 





Country 

Girls 

Boys 

Difference 

(Absolute 

Value) 

Gender Difference 

Percent of 
Students 

Average Scale 
Score 

Percent of 
Students 

Average Scale 
Score 

Girls Boys 

Scored Higher Scored Higher 

Ninth Grade Participants 

>K South Africa 

48(1.0) 

354 (3.0) 

52 (1.0) 

350 (3.4) 

3 (4.0) 

■ 

v Botswana 

51 (0.6) 

403 (2.9) 

49 (0.6) 

390 (3.0) 

14 (3.1) 


2 * Flonduras 

55 (1.0) 

328(4.1) 

45 (1.0) 

351 (4.1) 

23 (3.5) 



Benchmarking Participants 


2 Ontario, Canada 49(0.9) 512(2.7) 51(0.9) 512(3.1) 0(3.1) 








Quebec, Canada 51 (1.4) 531 (2.9) 49(1.4) 532 (2.5) 0 (2.7) 








1 Minnesota, US 52 (1.5) 545 (4.9) 48 (1.5) 545 (5.1) 0 (3.9) 








'Alabama. US 51 (1.6) 467 (6.3) 49(1.6) 465 (6.2) 2 (3.9) 








Abu Dhabi, UAE 47 (2.7) 450 (3.9) 53 (2.7) 448 (5.7) 2 (6.4) 








* Alberta, Canada 50(0.9) 504 (3.3) 50(0.9) 506 (2.7) 2 (3.0) 








i 3 North Carolina, US 52 (1.0) 535 (6.2) 48 (1.0) 539 (8.3) 3 (5.1) 




1 




' 2 California, US 49(1.1) 491 (5.6) 51 (1.1) 494 (5.0) 3 (4.1) 




1 




'Colorado, US 51 (1.5) 516(5.4) 49(1.5) 520 (5.0) 4(3.4) 




■ 




i 2 Connecticut, US 49(1.0) 520(5.2) 51 (1.0) 516 (5.4) 4 (4.5) 



■ 





i 2 Massachusetts, US 50 (1.0) 558 (6.0) 50 (1.0) 563 (5.5) 5 (4.5) 




■ 




' 2 Florida, US 49(1.9) 509(6.6) 51 (1.9) 517 (7.3) 8 (5.5) 




■ 




' 2 Indiana, US 52 (1.1) 518(5.1) 48(1.1) 526 (5.9) 8 (4.0) 




■ 




Dubai, UAE 48 (4.6) 486 (4.3) 52 (4.6) 470 (5.4) 16 (8.9) 









80 40 0 40 80 

■ Difference statistically significant 

■ Difference not statistically significant 


M Average achievement not reliably measured because the percentage of students with achievement too low for estimation exceeds 25%. 

V Reservations about reliability of average achievement because the percentage of students with achievement too low for estimation does not exceed 25% but exceeds 1 5%. 
See Appendix C.3 for target population coverage notes 1, 2, and 3. See Appendix C.9 for sampling guidelines and sampling participation notes t, 4, and I. 

( ) Standard errors appear in parentheses. Because of rounding some results may appear inconsistent 


TIMSS & PIRLS 

International Study Center 

Lynch School of Education, Boston College 


ANALYZING THE TIMSS 201 1 
INTERNATIONAL DATABASE USING SAS 

CHAPTER 3 61 


SOURCE: lEA'sTrendsin International Mathematics and Science Study -TIMSS 2011 





Exhibit 3.1 2: Example SAS Program for Student-level Analysis with Achievement by Gender (EXAMPLE2.SAS) 


LIBNAME Til "<datpath>" ; 

% INCLUDE " <macpath> JACKPV . SAS " ; 

DATA BSGALLM5 ; 

SET Til . BSGALLM5 ; 

WHERE NMISS (ITSEX) = 0 ; 

PROC FORMAT LIBRARY = WORK ; 

VALUE COUNTRY 

< list TIMSS 2011 country formats > ; 

VALUE SEX 

1 = "GIRL" 

2 = "BOY" ; 

%JACKPV ( TOTWGT , JKZONE, JKREP, 75, IDCNTRY ITSEX, BSMMATO , 5, BSGALLM5 ) ; 

PROC PRINT DATA = FINAL NOOBS ; 

VAR IDCNTRY ITSEX N TOTWGT MNPV MNPV_SE PCT PCT_SE ; 

FORMAT IDCNTRY COUNTRY. ITSEX SEX. N 6.0 TOTWGT 10.0 
MNPV MNPV_SE PCT PCT_SE 6.2 ; 


The results obtained from this program are shown in Exhibit 3. 13. For the sake of brevity, only 
the results of the first four countries are shown. The very same set of steps used in our first example 
apply here, adding ITSEX as a classification variable along with IDCNTRY. 

Exhibit 3.1 3: Output for Example Student-level Analysis with Achievement by Gender (EXAMPLE 2) 


IDCNTRY 

ITSEX 

N 

TOTWGT 

MNPV 

MNPV_SE 

PCT 

PCT_SE 

AUSTRALIA 

GIRL 

3747 

125555 

500.41 

4 . 72 

49.83 

1.61 

AUSTRALIA 

BOY 

3809 

126429 

509 . 16 

7.26 

50.17 

1.61 

BAHRAIN 

GIRL 

2288 

6276 

430 . 78 

2 . 51 

49.72 

0 . 85 

BAHRAIN 

BOY 

2352 

6348 

387 . 89 

3 . 07 

50.28 

0 . 85 

ARMENIA 

GIRL 

2894 

17702 

471.52 

3 . 07 

48.93 

0.78 

ARMENIA 

BOY 

2952 

18479 

461 . 86 

3.21 

51.07 

0.78 

CHILE 

GIRL 

3133 

133160 

409.46 

3.23 

53 . 01 

1.46 

CHILE 

BOY 

2702 

118020 

423 . 94 

3 . 05 

46.99 

1.46 


In Exhibit 3.13, each country’s results are displayed on two lines, one for each value of the 
variable ITSEX. The countries are identified in the first column and the second column describes the 
category of ITSEX being reported. The third column reports the number of valid cases and the fourth 
the sum of weights of the sampled students. The next four columns report the estimated average 
mathematics achievement and its standard error, followed by the percentage of students in each 
category and its standard error. As shown in the first two lines, the average mathematics achievement 
was 500.41 for eighth grade girls in Australia (standard error of 4.72) and 509.16 (standard error 
of 7.26) for eighth grade boys. An estimated 49.83 percent (standard error of 1.61) of students in 
Australia were girls, and 50.17 percent were boys (standard error of 1.61). 
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Student-level Analysis with a Contextual Scale 

TIMSS 2011 has innovated in its reporting of contextual data by creating contextual scales based on 
Rasch modeling. A good number of contextual scales were reported in the TIMSS 2011 
International Reports and are available in the International Database for analysis. Each contextual 
scale variable is a Rasch scale with an international mean of 10 and international standard deviation 
of 2. An index was derived from each scale that divides the range of scores into usually three 
categories: the most desirable scores (high values), the least desirable scores (low values), and the 
remaining scores in between. 

These contextual scales and their corresponding indices were reported in the TIMSS 2011 
International Reports. Exhibit 3.14 shows one such example, Exhibit 8.2 from TIMSS 2011 
International Results in Mathematics, reporting how much students like learning mathematics. 
Results on the Rasch scale are reported for each country as an “Average Scale Score” and its 
corresponding index is reported as the percentages of students in each category — Like Learning 
Mathematics, Somewhat Like Learning Mathematics, and Do Not Like Learning Mathematics — 
along with the average achievement in each category. 

This example will replicate both the average scale score of the Students Like Learning 
Mathematics contextual scale and the percentages of students, with their average mathematics 
achievement, in each category of its index. This will be done in two steps, both using the merged 
BSGALLM5 data fde. 

The first step, our third example, will compute the average scale score using the contextual scale 
variable BSBGSLM. After reviewing the appropriate codebook, we observe that the variable 
BSBGSLM in the student background data files contains infonnation on the Students Like Learning 
Mathematics contextual scale. We then proceed to read from the student background data files our 
variable of interest (BSBGSLM), the student sampling weight (TOTWGT), the variables that contain 
the jackknifing information (JKZONE and JKREP), and the country identification variable 
(IDCNTRY). Again, we will use the data of all available countries contained in the file BSGALLM5. 

The SAS program that implements our third example is presented in Exhibit 3.15 and is included 
in the International Database under the name EXAMPLE3.SAS. Note that one of the steps in this 
program is to select only those students who have non-missing data in our variable of interest 
BSBGSLM. 

The results obtained from this program are shown in Exhibit 3.16. Lor the sake of brevity, only 
the results of the first four countries are shown. 


3 The contextual scales are described in the context questionnaire scales section of Methods and Procedures in TIMSS and PIRLS 2011 (Martin & 
Mullis, 2012). 
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Exhibit 3.14: Example Exhibit of a Contextual Scale, Taken from 

TIMSS20 7 7 International Results in Mathematics (Exhibit 8.2) 


Exhibit 8.2: Students Like Learning Mathematics 


TIMSS 2011 
Mathematics 


8 th 

Grade 


Reported by Students 

Students were scored according to their degree of agreement with five statements on the Students Like Learning Mathematics scale. Students who 
Like Learning Mathematics had a score on the scale of at least 1 1 .3, which corresponds to their "agreeing a lot" with three of the five statements and 
"agreeing a little" with the other two, on average. Students who Do Not Like Learning Mathematics had a score no higher than 9.0, which corresponds 
to their "disagreeing a little" with three of the five statements and "agreeing a little" with the other two, on average. All other students Somewhat Like 
Learning Mathematics. 


Like Learning Somewhat Like Do Not Like 


Country 

Mathematics 

Learning Mathematics 

Learning Mathematics 


Average 


Percent 

Average 

Percent 

Average 

Percent 

Average 


Scale Score 



of Students 

Achievement 

of Students 

Achievement 

of Students 

Achievement 




Morocco 

48 (0.7) 

398 (2.4) 

40 (0.7) 

353 (2.2) 

12 (0.5) 

340 (4.6) 


11.2 (0.03) 


Armenia 

43 (1.0) 

499 (3.1) 

39 (0.8) 

451 (3.4) 

18 (1.0) 

437 (4.8) 


10.9 (0.05) 


Jordan 

42 (1.5) 

442 (3.7) 

39 (1.0) 

388 (4.2) 

19 (0.9) 

376 (4.8) 


10.9 (0.06) 


Georgia 

42 (1.3) 

463 (5.0) 

40 (1.0) 

423 (4.1) 

18 (1.0) 

405 (6.2) 

1 ■ 

10.8 (0.06) 

■ 

Malaysia 

39 (1.3) 

463 (5.0) 

46 (0.9) 

430 (5.6) 

15 (0.9) 

413(8.1) 


10.8 (0.05) 


Iran, Islamic Rep. of 

39 (1.1) 

450 (5.4) 

40 (0.8) 

3% (4.2) 

22 (0.9) 

388 (4.5) 

1 ■ 

10.6 (0.05) 

'1 

Ghana 

38 (1.4) 

370 (4.8) 

51 (1.2) 

314 (4.0) 

10 (0.5) 

299 (6.7) 


10.9 (0.05) 


Oman 

38 (0.8) 

420 (3.0) 

45 (0.8) 

342 (3.6) 

17 (0.7) 

324 (4.4) 


10.8 (0.04) 


Kazakhstan 

38 (1.5) 

506 (4.4) 

52 (1.3) 

478 (4.4) 

10 (0.7) 

475 (7.4) 


10.9 (0.05) 


Tunisia 

38 (1.0) 

448 (3.4) 

40 (0.8) 

415 (3.2) 

23 (0.9) 

405 (3.3) 

1 ■ 

10.6 (0.05) 

■ 

Syrian Arab Republic 

37 (1.1) 

408 (5.2) 

44 (1.0) 

373 (4.8) 

19 (0.9) 

353 (6.3) 


10.7 (0.05) 


Ukraine 

36 (1.7) 

502 (4.9) 

43 (1.21 

477 (4.1) 

20 (1.2) 

450 (4.9) 

1 ■ 

10.6 (0.07) 

i 

Lebanon 

35 (1.2) 

475 (4.6) 

43 (1.0) 

441 (4.2) 

21 (1.1) 

425 (5.6) 


10.6 (0.06) 


Singapore 

32 (0.7) 

637 (3.9) 

44 (0.7) 

610 (4.1) 

23 (0.7) 

578 (4.4) 


10.4 (0.03) 


Turkey 

31 (1.0) 

504 (6.0) 

42 (0.7) 

436 (3.9) 

26 (1.0) 

420 (3.5) 


10.3 (0.05) 


United Arab Emirates 

31 (0.7) 

488 (2.3) 

42 (0.6) 

448 (2.5) 

27 (0.8) 

432 (2.5) 

1 ■ 

10.2 (0.04) 

□ 

Palestinian Nat'l Auth. 

31 (1.1) 

447 (5.0) 

43 (1.0) 

394 (4.1) 

26 (1.1) 

375 (5.1) 


10.3 (0.05) 


Russian Federation 

29 (1.1) 

567 (4.7) 

49 (0.9) 

537 (3.6) 

22 (1.0) 

509 (4.1) 

1 ■ 

10.3 (0.04) 

■ 

Saudi Arabia 

29 (1.3) 

436 (5.6) 

40 (1.0) 

389 (5.4) 

32(1.4) 

364 (4.1) 


10.1 (0.07) 


Qatar 

27 (1.0) 

456 (4.5) 

43 (0.8) 

401 (3.7) 

31 (1.2) 

386 (4.8) 

1 ■ 

10.0 (0.05) 

■ 

Thailand 

26 (1.1) 

456 (5.6) 

57 (0.9) 

420 (4.5) 

16 (1.0) 

408 (5.1) 


10.3 (0.05) 


Israel 

26 (0.8) 

536 (5.1) 

40 (0.7) 

523 (4.3) 

35 (1.0) 

496 (5.1) 


9.9 (0.04) 


Macedonia, Rep. of 

24 (1.0) 

462 (6.2) 

40 (1.0) 

422 (6.3) 

36(1.4) 

425 (6.0) 


9.8 (0.06) 


Bahrain 

24 (0.6) 

454 (4.6) 

38 (0.9) 

413 (2.7) 

38 (0.8) 

381 (3.4) 

1 ■ 

9.8 (0.03) 

■ 

Lithuania 

22 (1.0) 

531 (3.71 

44 (1.0) 

506 (2.7) 

34 (1.1) 

482 (3.3) 


9.8 (0.05) 


Chile 

22 (0.9) 

449 (3 j) 

40 (0.9) 

416 (2.9) 

38 (1.0) 

398 (2.9) 

1 ■ 

9.8 (0.04) 

■ 

Indonesia 

20 (1.4) 

396 (6.1) 

70(1.2) 

385 (4.5) 

10 (0.8) 

382 (6.3) 


10.4 (0.04) 


United States 

19 (0.6) 

536 (3.2) 

40 (0.6) 

515 (3.0) 

40 (0.8) 

494 (2.8) 

1 ■ 

9.5 (0.04) 

■ 

Hong Kong SAR 

19 (0.8) 

635 (4.4) 

44 (1.0) 

595 (3.8) 

37(1.3) 

551 (4.6) 


9.6 (0.05) 


Romania 

18(1.0) 

516 (6.1) 

40 (1.0) 

459 (4.3) 

41 (1.2) 

438 (4.8) 

1 ■ 

9.5 (0.05) 

■ 

Italy 

18 (0.9) 

538 (3.6) 

42 (0.9) 

507 (2.8) 

40 (1.3) 

472 (3.2) 


9.6 (0.05) 


New Zealand 

17 (1.0) 

525 (6.9) 

41 (1.0) 

497 (5.7) 

42 (1.5) 

467 (4.8) 

1 ■ 

9.5 (0.06) 

■ 

Norway 

17 (0.9) 

511 (4.1) 

42 (1.0) 

482 (2.6) 

42 (1.4) 

453 (2.8) 


9.4 (0.05) 


Australia 

16 (0.9) 

553 (7.5) 

40 (0.9) 

520 (5.6) 

45 (1.4) 

476 (4.4) 


9.3 (0.06) 


Hungary 

15 (0.7) 

549 (5.6) 

35 (1.0) 

508 (4.8) 

50 (1.3) 

491 (3.8) 


9.2 (0.05) 


England 

14(1.0) 

548 (8.9) 

44 (1.3) 

517 (5.7) 

42 (1.7) 

484 (5.2) 

1 ■ 

9.4 (0.07) 

i 

Chinese Taipei 

14 (0.7) 

681 (4.3) 

33 (0.9) 

645 (3.6) 

53 (U) 

568 (3.2) 


9.0 (0.06) 


Sweden 

13 (0.6) 

524 (4.0) 

42 (0.7) 

498(1.8) 

44 (1.0) 

462 (2.1) 

1 ■ 

9.4 (0.04) 

i 

Finland 

10 (0.6) 

560 (4.1) 

34 (1.0) 

532 (2.8) 

57(1.1) 

496 (2.6) 


8.8 (0.05) 


Japan 

9 (0.6) 

621 (5.1) 

38(1.1) 

589 (3.3) 

53 (1.4) 

545 (3.1) 

1 ■ 

9.1 (0.05) 

Korea, Rep. of 

8 (0.3) 

677 (4.7) 

36 (0.7) 

649 (3.3) 

56 (0.8) 

581 (2.9) 


8.9 (0.03) 


Slovenia 

6 (0.4) 

544 (5.3) 

31 (1.1) 

521 (3.0) 

63 (1.3) 

494 (2.4) 

1 ■ 

8.6 (0.05) 

■ 

I International Avg. 

26 (02) 

504 (0.8) 

42 (0.1) 

467 (0.6) 

31 (02) 

443 (0.7) 

1 



Ninth Grade Participants 










Botswana 

47(1.1) 

427 (2.5) 

38 (0.8) 

376 (2.6) 

16 (0.8) 

370 (4.3) 


11.0 (0.05) 


South Africa 

41 (0.9) 

378 (2.0) 

44 (0.7) 

339 (2.9) 

15 (0.6) 

348 (5.3) 

1 ■ 

10.8 (0.04) 

1 

Honduras 

23 (1.0) 

364 (4.6) 

49 (0.9) 

332 (4.2) 

28(1.1) 

334 (4.8) 


10.1 (0.06) 


Benchmarking Participants 










Abu Dhabi, UAE 

32 (1.2) 

485 (4.4) 

42 (1.0) 

441 (3.6) 

26 (1.4) 

420 (4.9) 


10.3 (0.06) 


Dubai, UAE 

29(1.0) 

508 (3.5) 

41 (0.9) 

473 (3.1) 

30 (1.0) 

456 (3.1) 

1 ■ 

10.1 (0.05) 

■ 

Ontario, Canada 

26(1.1) 

546 (3.5) 

41 (1.0) 

513 (3.4) 

34 (1.4) 

481 (3.0) 


9.9 (0.06) 


North Carolina, US 

24(1.8) 

556 (7.6) 

44(1.1) 

542 (7.8) 

31 (2.3) 

516 (7.0) 

9.9 (0.11) 

■ 

Connecticut, US 

22 (1.5) 

552 (6.0) 

40(1.2) 

526 (5.2) 

38 (1.8) 

495 (5.4) 


9.7 (0.08) 


Colorado, US 

20 (1.6) 

548 (5.9) 

38(1.7) 

528 (4.8) 

42 (2.1) 

495 (5.8) 

1 ■ 

9.4 (0.10) 

■ 

Massachusetts, US 

19 (1.3) 

585 (6.1) 

40 (1.0) 

568 (5.4) 

41 (1.7) 

543 (5.4) 


9.4 (0.09) 


Minnesota, US 

18(1.5) 

578 (6.8) 

41 (0.9) 

555 (4.7) 

41 (1.6) 

521 (4.6) 


9.5 (0.08) 


Alabama, US 

18 (1.9) 

475 (10.7) 

37 (0.9) 

471 (6.7) 

45 (1.7) 

460 (5.3) 


9.3 (0.11) 


Florida, US 

17(1.1) 

552 (9.7) 

38(1.4) 

525 (6.9) 

45(1.7) 

493 (6.2) 


9.4 (0.08) 


California, US 

17 (0.9) 

519 (6.4) 

42 (1.3) 

496 (6.1) 

41 (1.8) 

480 (5.0) 


9.4 (0.07) 


Alberta, Canada 

16 (0.9) 

531 (4.7) 

44 (1.0) 

514 (2.5) 

40 (1.4) 

486 (3.1) 

1 ■ 

9.4 (0.06) 

■ 

Indiana, US 

16 (1.4) 

547 (6.2) 

39 (1.3) 

529 (5.3) 

45 (2.0) 

507 (5.0) 


9.3 (0.10) 


Quebec, Canada 

12 (0.7) 

557 (3.9) 

43 (0.9) 

540 (2.4) 

44 (1.2) 

517 (2.6) 

1 ■ 

9.3 (0.05) 

■ 


■T 

•o 

& 


I 

I 

1 

5 

i 


I 

£ 

S 

c 


Centerpoint of scale set at 1 0. 

( ) Standard errors appear in parentheses. Because of rounding some results may appear Inconsistent. 
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Exhibit 3.1 5: Example SAS Program for Student-level Analysis with a Contextual Scale (EXAMPLE3.SAS) 


LIBNAME Til "<datpath>" ; 

% INCLUDE " <macpath> JACKGEN . SAS " ; 

DATA BSGALLM5 ; 

SET Til . BSGALLM5 ; 

WHERE NMISS (BSBGSLM) = 0 ; 

PROC FORMAT LIBRARY = WORK ; 

VALUE COUNTRY 

< list TIMSS 2011 country formats > ; 

% JACKGEN ( TOTWGT , JKZONE, JKREP, 75, IDCNTRY, BSBGSLM, BSGALLM5 ) ; 

PROC PRINT DATA = FINAL NOOBS ; 

VAR IDCNTRY N TOTWGT MNX MNX_SE PCT PCT_SE ; 

FORMAT IDCNTRY COUNTRY. N 6.0 TOTWGT 10.0 
MNX MNX_SE PCT PCT_SE 6.2 ; 


In the first line of the results in Exhibit 3.16, we see that students in Australia scored 9.32, with a 
standard error of 0.06, on the Students Like Learning Mathematics contextual scale. Note that this is 
below the international average of 10, the difference being statistically significant at the 95% 
confidence level. 

Exhibit 3.1 6: Output for Example Student-level Analysis with a Contextual Scale (EXAMPLE 3) 


IDCNTRY 

N 

TOTWGT 

MNX 

MNX_SE 

PCT 

PCT_SE 

AUSTRALIA 

7389 

248762 

9.32 

0 . 06 

1.19 

0 . 02 

BAHRAIN 

4581 

12458 

9 . 77 

0 . 03 

0 . 06 

0 . 00 

ARMENIA 

5626 

34775 

10.87 

0 . 05 

0.17 

0 . 00 

CHILE 

5772 

248321 

9 . 76 

0 . 04 

1.18 

0 . 03 


In the second step, our fourth example, we will compute the percentages of eighth grade students, 
with their average mathematics achievement, in each category of the index variable BSDGSLM. This 
analysis is similar to our earlier example of a student-level analysis with achievement by gender. Our 
variable of interest here is the BSDGSLM index variable of our contextual scale and it will be used 
to measure average mathematics achievement using the five plausible values BSMMAT01 through 
BSMMAT05. 

The SAS program that implements this fourth example is presented in Exhibit 3.17 and is 
included in the International Database under the name EXAMPLE4.SAS. Note that one of the steps 
in this program is to select only those students who have non-missing data in our variable of interest 
BSDGSLM. The results obtained from this program are shown in Exhibit 3.18. For the sake of 
brevity, only the results of the first four countries are shown. 

As shown in the first three lines of the results, 15.67 percent of students in Australia liked 
learning mathematics (standard error of 0.94), and their average mathematics achievement 
was 553.21 (standard error of 7.47); 39.81 percent somewhat liked learning mathematics (standard 
error of 0.87), and their average mathematics achievement was 519.53 (standard error of 5.58); and 
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44.53 percent of students did not like learning mathematics (standard error of 1.41), and their 
average mathematics achievement was 475.97 (standard error of 4.43). 

Exhibit 3.1 7 Example SAS Program for Student-level Analysis with a Contextual Scale (EXAMPLE4.SAS) 


LIBNAME Til "<datpath>" ; 

% INCLUDE " cmacpat h> JACKPV . SAS " ; 

DATA BSGALLM5 ; 

SET Til . BSGALLM5 ; 

WHERE NMISS (BSDGSLM) = 0 ; 

PROC FORMAT LIBRARY = WORK ; 

VALUE COUNTRY 

< list TIMSS 2011 country formats > ; 

VALUE SLM 

1 = "LIKE LEARNING MATHEMATICS" 

2 = "SOMEWHAT LIKE LEARNING MATHEMATICS" 

3 = "DO NOT LIKE LEARNING MATHEMATICS" ; 

% JACKPV (TOTWGT, JKZONE, JKREP, 75, IDCNTRY BSDGSLM, BSMMATO , 5, BSGALLM5 ) ; 

PROC PRINT DATA = FINAL NOOBS ; 

VAR IDCNTRY BSDGSLM N TOTWGT MNPV MNPV_SE PCT PCT_SE ; 

FORMAT IDCNTRY COUNTRY. BSDGSLM SLM. N 6.0 TOTWGT 10.0 
MNPV MNPV_SE PCT PCT_SE 6.2 ; 


Exhibit 3.1 8 Output for Example Student-level Analysis with a Contextual Scale (EXAMPLE 4) 


IDCNTRY 

BSDGSLM 

N 

TOTWGT 

MNPV 

MNPV_SE 

PCT 

PCT_SE 

AUSTRALIA 

LIKE LEARNING MATHEMATICS 

1068 

38977 

553.21 

7.47 

15 . 67 

0 . 94 

AUSTRALIA 

SOMEWHAT LIKE LEARNING MATHEMATICS 

2985 

99021 

519.53 

5.58 

39 . 81 

0 . 87 

AUSTRALIA 

DO NOT LIKE LEARNING MATHEMATICS 

3336 

110764 

475 . 97 

4.43 

44.53 

1.41 

BAHRAIN 

LIKE LEARNING MATHEMATICS 

1072 

2959 

454.13 

4.64 

23 . 75 

0 . 64 

BAHRAIN 

SOMEWHAT LIKE LEARNING MATHEMATICS 

1756 

4780 

413.45 

2.71 

38.37 

0 . 86 

BAHRAIN 

DO NOT LIKE LEARNING MATHEMATICS 

1753 

4719 

380.58 

3.37 

37.88 

0 . 84 

ARMENIA 

LIKE LEARNING MATHEMATICS 

2421 

14925 

499.36 

3 . 07 

42 . 92 

0 . 97 

ARMENIA 

SOMEWHAT LIKE LEARNING MATHEMATICS 

2181 

13729 

451.39 

3.37 

39.48 

0 . 76 

ARMENIA 

DO NOT LIKE LEARNING MATHEMATICS 

1024 

6121 

436 . 88 

4.75 

17.60 

0 . 97 

CHILE 

LIKE LEARNING MATHEMATICS 

1289 

54770 

449.31 

3.52 

22 . 06 

0 . 86 

CHILE 

SOMEWHAT LIKE LEARNING MATHEMATICS 

2291 

99860 

416.42 

2 . 93 

40.21 

0 . 89 

CHILE 

DO NOT LIKE LEARNING MATHEMATICS 

2192 

93691 

398.30 

2 . 95 

37.73 

0 . 97 


3.8 TIMSS Analyses with Teacher-level Variables 

The teachers in the TIMSS 2011 International Database do not constitute representative samples of 
teachers in the participating countries. Rather, they are the teachers of nationally representative 
samples of students. Therefore, analyses with teacher data should be made with students as the units 
of analysis and reported in terms of students who are taught by teachers with a particular attribute. 

When analyzing teacher data, it is first necessary to link the students to their respective teachers. 
The student-teacher linkage data files (AST/BST) were created for this purpose. Student 
achievement scores (plausible values), jackknife replication information, and teacher weighting 
variables — MATWGT for mathematics teachers, SCIWGT for science teachers, or TCHWGT for all 
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teachers — appropriate for conducting analyses with teacher variables are found in the 
student-teacher linkage data files in order to simplify the merging process for analyses that li nk 
teacher background variables to student achievement. For such analyses, it only is necessary to 
merge the teacher background data files (ATG/BTM/BTS) with the student-teacher linkage data 
files. For analyses linking teacher variables to student background variables, it also is necessary to 
merge the student background data files (ASG/BSG) with the teacher background data files 
combining the latter with the student-teacher linkage data files. 

As our example of an analysis using teacher background data, we will investigate the years of 
experience of the TIMSS 2011 eighth grade mathematics teachers. The results of such an analysis are 
presented in Exhibit 7.6 of TIMSS 2011 International Results in Mathematics and are reproduced 
here in Exhibit 3.19. 

Conducting analyses with teacher data requires a few extra steps. As before, we first proceed to 
identify the variables relevant to the analysis in the appropriate files, and review the documentation 
for any specific national adaptations to the questions of interest (Supplements 1 and 2 to this User 
Guide). Because we are using a teacher-level variable, we must use the teacher background data files 
and the student-teacher linkage data files to find the variables. From the teacher background data 
files, we need the variable that contains the information on the eighth grade mathematics teachers’ 
years of experience (BTDG01), the variable that identifies the country (IDCNTRY), and the two 
teacher identification variables (IDTEACH and IDLINK) that will allow us to link the teacher data to 
the student data. 4 

We then proceed to retrieve the necessary information from the eighth grade student-teacher 
linkage data files. From these files, we need the country identification (IDCNTRY) and the two 
teacher identification variables (IDTEACH and IDLINK) to link the teacher data to the student data. 
We also need the jackknife replication variables (JKZONE and JKREP), the mathematics teacher 
weighting variable (MATWGT), and the eighth grade mathematics achievement plausible values 
(BSMMAT01 through BSMMAT05). 

In general, to perform analyses using the teacher background data files, users should do the 
following: 

1 . Identify the variables of interest in the teacher background data files and note any specific 
national adaptations to the variables. 

2. Retrieve the relevant variables from the teacher background data files, including analysis 
variables, classification variables, identification variables (IDCNTRY, IDTEACH, and IDLINK), 
and any other variables used in the selection of cases. 

3. Retrieve the relevant variables from the student-teacher linkage data files, including plausible 
values of achievement, identification variables (IDCNTRY, IDSTUD, IDTEACH, and IDLINK), 
sampling (JKZONE and JKREP) and weighting (MATWGT, SCIWGT, or TCHWGT) variables, 
and any other variables used in the selection of cases. 


4 The information about teachers’ years of experience found in the BTBG01 variable was collapsed into reporting categories and stored in the 
derived variable BTDG01 (see Supplement 3 to this User Guide). 
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Exhibit 3.19: Exhibit of Example Teacher-level Analysis, Taken from 

TIMSS 2011 International Results in Mathematics (Exhibit 7.6) 


Exhibit 7.6: Teachers'Years of Experience 

Reported by Teachers 






TIMSS 2011 Eg 
Mathematics 

Country 

20 Years or More 

At Least 10 but Less 
than 20 Years 

At Least 5 but Less 
than 10 Years 

Less than 5 Years 

Average 
Years of 
Experience 

Percent of 
Students 

Average 

Achievement 

Percent of 
Students 

Average 

Achievement 

Percent of 
Students 


Average 

Achievement 

Percent of 
Students 

Average 

Achievement 

Armenia 

63 (3.7) 

467 (3.9) 

30 (3.3) 

464 (6.0) 

4 (1.6) 


473 (24.9) 

3 (1.4) 

474 (18.4) 

24 (0.8) 

Australia r 

37 (4.0) 

519 (8.1) 

22 (3.4) 

513 (10.8) 

18 (3.2) 

II 

504 (17.1) 

24 (3.4) 

485 (8.4) 

15 (0.9) 

Bahrain 

19 (2.2) 

433 (7.0) 

54 (3.6) 

404 (3.7) 

17 (2.7) 


403 (5.8) 

10 (1.9) 

430 (9.1) 

14 (0.4) 

Chile 

49 (3.8) 

415 (4.6) 

15 (2.9) 

416 (10.0) 

13 (2.8) 


421 (12.1) 

22 (3.4) 

421 (6.3) 

19 (1.0) 

Chinese Taipei 

24 (3.6) 

621 (7.2) 

41 (3.6) 

607 (5.8) 

26 (3.5) 


608 (9.3) 

9 (2.5) 

593 (8.9) 

14 (0.7) 

England 

21 (3.6) 

510(15.5) 

25(3.7) 

516(11.8) 

22 (3.9) 


495 (11.6) 

32 (3.9) 

503 (10.7) 

12 (0.9) 

Finland 

41 (3.4) 

517 (2.8) 

27 (3.4) 

511 (5.3) 

18 (2.8) 


515 (6.1) 

15 (2.4) 

510 (5.2) 

16 (0.7) 

Georgia 

63 (3.9) 

428(5.2) 

21 (3.5) 

441 (10.1) 

9(2.4) 

II 

439 (15.0) 

7 (2.3) 

431 (18.5) 

25 (1.1) 

Ghana 

6 (1.8) 

360 (19.9) 

23 (3.8) 

340 (9.0) 

28 (4.0) 


334 (9.3) 

43 (3.9) 

321 (6.8) 

8 (0.5) 

Hong Kong SAR 

18 (3.3) 

570(11.9) 

39 (4.3) 

590 (8.4) 

25 (4.2) 


589 (11.9) 

18 (3.3) 

588 (10.1) 

12 (0.7) 

Hungary 

62 (3.5) 

508 (4.4) 

26 (3.0) 

508 (6.2) 

7 (1.9) 


488 (18.6) 

5 (1.5) 

456 (21.5) 

22 (0.7) 

Indonesia 

25 (3.9) 

402 (9.1) 

30 (4.0) 

399 (9.1) 

19 (3.3) 

II 

385 (8.0) 

26 (4.5) 

356(9.1) 

13 (0.8) 

Iran, Islamic Rep. of 

28 (3.2) 

443 (8.9) 

40 (3.8) 

416 (6.0) 

16 (2.6) 


402 (10.4) 

16 (2.8) 

374 (10.7) 

14 (0.6) 

Israel 

38 (2.8) 

545 (6.6) 

36 (2.8) 

518 (6.6) 

15 (2.0) 


495 (10.7) 

11 (1.8) 

468 (14.4) 

17 (0.5) 

Italy 

60 (4.1) 

502 (3.2) 

22 (3.3) 

492 (7.3) 

11 (2.5) 


504 (9.1) 

8 (2.1) 

492 (13.6) 

22 (0.9) 

Japan 

47 (3.9) 

576 (3.7) 

18 (3.1) 

558 (5.5) 

17 (2.3) 


575 (9.1) 

18 (3.1) 

559(7.5) 

17 (0.8) 

Jordan 

16 (2.6) 

406 (8.5) 

29 (3.3) 

410 (7.6) 

29 (3.5) 


394 (9.6) 

26 (3.1) 

413 (7.0) 

11 (0.6) 

Kazakhstan 

62 (3.9) 

492 (5.2) 

21 (3.2) 

468 (8.6) 

9 (2.7) 


489 (14.9) 

8 (2.2) 

493 (14.8) 

22 (0.9) 

Korea, Rep. of 

34(3.1) 

618 (5.0) 

22 (2.8) 

616 (8.8) 

17 (2.1) 


625 (7.1) 

27 (2.6) 

594 (4.8) 

13 (0.6) 

Lebanon 

27 (3.6) 

454 (7.9) 

32 (3.9) 

445 (6.9) 

21 (3.2) 

II 

460 (9.8) 

20(3.5) 

445 (8.7) 

14 (1.0) 

Lithuania 

73 (3.4) 

501 (3.0) 

17 (2.6) 

509 (6.8) 

7 (2.1) 


504 (19.6) 

3 (1.4) 

506 (17.8) 

25 (0.8) 

Macedonia, Rep. of r 

50 (4.4) 

421 (9.1) 

25 (4.2) 

430 (12.0) 

12 (2.7) 


415 (15.3) 

13 (2.9) 

420 (18.6) 

20 (0.9) 

Malaysia 

18 (3.0) 

446 (12.2) 

31 (3.4) 

446 (9.5) 

21 (3.0) 


426 (11.4) 

30 (3.3) 

441 (10.5) 

11 (0.7) 

Morocco 

69 (2.8) 

374 (2.8) 

11 (2.0) 

373 (9.0) 

5 (1.5) 


358 (12.2) 

15 (2.3) 

363 (6.3) 

22 (0.6) 

New Zealand 

36 (3.0) 

492 (8.4) 

22(2.7) 

486 (9.6) 

25 (3.0) 


489 (8.9) 

17 (2.8) 

482 (15.6) 

15 (0.8) 

Norway 

30 (4.0) 

478 (3.7) 

25 (3.6) 

474 (5.5) 

19(3.7) 

II 

475 (4.4) 

26 (3.5) 

474 (4.0) 

15(1.0) 

Oman 

7 (1.3) 

362 (12.2) 

25 (2.6) 

385 (6.5) 

46 (3.3) 


363 (4.7) 

21 (2.6) 

360 (6.9) 

9 (0.3) 

Palestinian Nat'l Auth. 

14(3.1) 

413(11.9) 

37 (3.9) 

410 (7.3) 

24 (3.6) 


400 (7.6) 

25 (3.2) 

394 (7.5) 

11 (0.7) 

Qatar 

23 (4.2) 

432 (12.7) 

36 (4.6) 

425 (9.4) 

25 (3.4) 


388 (9.2) 

16 (2.9) 

386 (10.1) 

13 (0.7) 

Romania 

66 (3.7) 

466 (5.2) 

24 (3.3) 

449 (9.3) 

6(1.7) 

li 

420 (15.9) 

4(1.6) 

423 (12.7) 

25 (0.9) 

Russian Federation 

67 (3.3) 

540 (4.4) 

24(3.1) 

543 (7.0) 

5 (1.2) 


515 (15.2) 

4 (1.2) 

547 (23.5) 

24 (0.6) 

Saudi Arabia 

13 (2.9) 

386 (10.2) 

41 (3.9) 

406 (7.3) 

25 (3.5) 


402 (8.9) 

21 (3.5) 

367 (7.7) 

11 (0.6) 

Singapore 

10 (1.4) 

618 (10.6) 

16 (2.1) 

619 (9.3) 

26 (2.4) 


624 (7.3) 

47 (2.5) 

601 (5.0) 

8 (0.4) 

Slovenia 

52 (2.9) 

506 (3.2) 

20 (2.6) 

500 (5.0) 

17 (2.0) 


500 (4.1) 

12 (1.9) 

515 (4.9) 

19 (0.6) 

Sweden r 

26 (2.7) 

486 (5.4) 

42 (3.4) 

489 (3.9) 

22 (2.7) 


482 (3.7) 

10 (2.0) 

476 (5.1) 

15(0.6) 

Syrian Arab Republic 

16(3.1) 

400 (9.6) 

26 (3.7) 

375 (7.9) 

24 (3.6) 


370 (8.8) 

35 (4.0) 

378 (8.7) 

10 (0.6) | 

Thailand 

34 (3.4) 

444 (8.4) 

21 (3.1) 

432 (11.0) 

18 (2.7) 


417 (11.6) 

28 (3.2) 

415 (8.7) 

15(0.8) 

Tunisia 

38 (3.3) 

442 (5.6) 

35 (3.3) 

419 (5.4) 

18 (2.8) 

II 

417 (7.5) 

10(2.1) 

394 (7.2) 

16 (0.7) 

Turkey 

11(2.2) 

471 (14.5) 

24 (3.2) 

481 (10.8) 

38 (3.5) 


445 (6.9) 

27 (2.8) 

431 (6.5) 

9 (0.5) 

Ukraine 

68 (4.4) 

477 (4.5) 

20 (3.6) 

491 (10.0) 

9(2.5) 


473 (11.1) 

3 (1.4) 

473 (18.7) 

25 (1.0) 

United Arab Emirates 

24 (2.0) 

442 (6.4) 

36 (2.4) 

455 (4.0) 

26 (2.3) 


461 (4.8) 

14 (1.8) 

467 (6.8) 

13 (0.4) 

United States r 

26 (2.2) 

519 (6.8) 

28 (2.4) 

517 (5.1) 

28 (2.8) 


506 (7.2) 

17 (2.2) 

505 (6.7) 

14 (0.6) 

I International Avg. 

36 (0.5) 

474 (1.3) 

28 (05) 

470 (1.2) 

19 (0.4) 


463 (1.7) 

18 (0.4) 

458 (1.8) 

16 (0.1) 1 

Ninth Grade Participants 

Botswana 

2 (1.0) 


39 (4.5) 

401 (5.3) 

31 (4.3) 


403 (4.2) 

29 (3.9) 

384 (5.2) 

9 (0.4) 

Honduras r 

26 (3.8) 

341 (6.5) 

23 (4.2) 

335 (10.8) 

22 (4.4) 


332 (8.4) 

29 (4.2) 

339 (11.1) 

12 (0.9) 

South Africa 

30 (3.8) 

344 (7.3) 

33 (3.4) 

358 (5.8) 

18 (3.0) 


364 (8.6) 

19 (3.1) 

345 (8.7) 

14 (0.8) 

Benchmarking Participants 

Alberta, Canada 

25 (3.5) 

506 (5.0) 

37 (4.3) 

504 (3.8) 

15 (3.0) 


504 (6.9) 

23 (3.4) 

505 (5.3) 

13 (0.7) 

Ontario, Canada 

16 (2.8) 

511 (7.5) 

44 (4.2) 

512 (3.8) 

31 (3.5) 


516 (4.9) 

10 (2.5) 

511 (9.4) 

12 (0.5) 

Quebec, Canada 

19 (3.0) 

544 (6.6) 

47 (3.8) 

536 (4.2) 

22 (3.2) 


524 (7.0) 

12 (2.6) 

521 (7.3) 

13 (0.6) 

Abu Dhabi, UAE 

25 (4.1) 

456 (14.3) 

30 (4.1) 

433 (6.3) 

29 (4.6) 


456 (8.5) 

16 (3.2) 

463 (9.2) 

14 (0.9) 

Dubai, UAE 

19 (2.2) 

443 (9.5) 

42 (2.6) 

491 (5.0) 

25 (3.3) 


488 (8.7) 

13 (2.6) 

471 (13.9) 

13 (0.5) 

Alabama, US i 

16 (4.8) 

494 (20.4) 

35 (7.8) 

473 (11.2) 

32 (6.2) 


450 (12.0) 

17(5.7) 

464 (11.2) 

12(1.3) 

California, US r 

19 (5.4) 

502 (25.5) 

33 (6.9) 

490 (9.2) 

28 (6.4) 


506 (10.2) 

20 (5.9) 

479 (21.5) 

12(1.3) 

Colorado, US r 

21 (4.9) 

564 (9.3) 

32 (5.6) 

If? 517(113) 

32 (5.81 


508 (14.0) 

15 (3.5) 

471 (13.3) 

13 (1.0) 

Connecticut, US 

29 (6.2) 

531 (17.9) 

32 (5.6) 

533 (9.2) 

20 (4.8) 


509 (18.9) 

19 (5.5) 

503 (14.5) 

14 (1.3) 

Florida, US r 

18 (5.3) 

530 (13.7) 

43 (7.0) 

521 (10.5) 

26 (5.81 


514(14.6) 

13 (4.0) 

524 (29.0) 

13 (1.2) 

Indiana, US r 

34(5.6) 

526 (11.0) 

22 (5.8) 

533 (13.8) 

27 (6.0) 


516(12.2) 

17 (5.2) 

494 (9.9) 

15 (1.4) 

Massachusetts, US 

10 (4.1) 

566 (20.3) 

33 (5.8) 

569 (10.9) 

39 (5.2) 

II 

552 (8.5) 

18 (5.5) 

556 (17.9) 

11 (1.3) 

Minnesota, US 

27 (6.4) 

556 (9.3) 

36 (5.2) 

553 (8.9) 

22 (4.5) 


531 (15.3) 

15 (4.3) 

528 (17.9) 

15(1.5) 

North Carolina, US r 

26 (5.5) 

559 (13.2) 

30 (5.6) 

530 (14.8) 

33 (5.5) 


545 (13.2) 

11 (4.3) 

517 (12.7) 

14(1.0) 


( ) Standard errors appear In parentheses. Because of rounding some results may appear inconsistent. 
A tilde (~) indicates Insufficient data to report achievement. 

An "r" indicates data are available for at least 70% but less than 85% of the students. 
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4. Merge the teacher background data files with the student-teacher linkage data files using the 
variables IDCNTRY, IDTEACH, and IDLINK. 

5. If student background variables also are needed, merge the student background data files with the 
merged student-teacher data files from the previous step using the variables IDCNTRY and 
IDSTUD. 

6. Perform any necessary variable transformations or recodes. 

7. Use the macros JACKGEN and JACKREG, or JACKPV and JACKREGP if plausible values are 
involved, with the appropriate parameters. 

8. Specify the location of the data files (<datpath>) and the macros (<macpath>). 

9. Print the results file. 

The two file types are merged and the resulting merged file is then input to the JACKPV macro. 
The merging is done using the combination of identification variables IDCNTRY, IDTEACH, and 
IDLINK. The combination of values for these three variables is unique within the teacher 
background data files, but is repeated in the student-teacher linkage data files as many times as 
needed to link a teacher to all students in a classroom. After the files are merged, the JACKPV macro 
is invoked and the results can be printed. 

For this analysis, we will again use the data for all available countries, making use of an 
aggregated teacher background data file, BTMALLM5, and an aggregated student-teacher linkage 
data file, BSTALLM5. These aggregated files were created with the JOIN macro. 

The SAS program that executes this example is presented in Exhibit 3.20 and is included in the 
International Database under the name EXAMPLE5.SAS. Note that one of the steps in this program 
is to select only those students who have non-missing data in our variable of interest BTDG01. The 
results obtained from this program are displayed in Exhibit 3.21, edited to show only the first four 
countries for the sake of brevity. 

Because our example uses data from mathematics teachers, the weighting variable MATWGT is 
specified. Analyses with science teachers require that SCIWGT be specified. For analyses with all 
teachers, TCHWGT should be specified. (See Chapter 4 for more information on sampling weights.) 

Each country’s results are displayed on four lines, one for each value of the BTDG01 variable. 
The results are presented in the same manner as in the previous examples, with countries identified 
in the first column, and the second column describing the categories of the analysis variable. As 
shown in the first four lines of results, 36.54 percent (standard error of 4.03) of eighth grade students 
in Australia were taught by teachers with 20 years or more of experience, 21.51 percent (standard 
error of 3.39) by teachers with 10 to 20 years of experience, 17.73 percent (standard error of 3.18) by 
teachers with 5 to 9 years of experience, and 24.22 percent (standard error of 3.38) by teachers with 
less than 5 years of experience. Also, the estimated average mathematics achievement was 519.11 
(standard error of 8.07) for eighth grade students taught by teachers with 20 years or more of 
experience, 5 13.05 (standard error of 10.77) for students taught by teachers with 10 to 20 years of 
experience, 503.68 (standard error of 17.14) for students taught by teachers with 5 to 9 years of 
experience, and 485.34 (standard error of 8.45) for students taught by teachers with less than 5 years 
of experience. 
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Exhibit 3.20: Example SAS Program for Teacher-level Analysis (EXAMPLE5.SAS) 


LIBNAME Til "<datpath>" ; 

% INCLUDE " <macpath> JACKPV . SAS " ; 

PROC SORT DATA = Til . BTMALLM5 OUT = BTMALLM5 ; 

BY IDCNTRY IDTEACH IDLINK ; 

PROC SORT DATA = Til . BSTALLM5 OUT = BSTALLM5 ; 

BY IDCNTRY IDTEACH IDLINK ; 

DATA MERGED ; 

MERGE BTMALLM5 (IN = INBTM) 

BSTALLM5 (IN = INBST) ; 

BY IDCNTRY IDTEACH IDLINK ; 

IF INBTM AND INBST ; 

DATA MERGED ; 

SET MERGED ; 

WHERE NMISS (BTDG01) = 0 ; 

PROC FORMAT LIBRARY = WORK ; 

VALUE COUNTRY 

< list TIMSS 2011 country formats > ; 

VALUE BTDG 

1 = "20 YEARS OR MORE" 

2 = "AT LEAST 10 BUT LESS THAN 20 YEARS" 

3 = "AT LEAST 5 BUT LESS THAN 10 YEARS" 

4 = "LESS THAN 5 YEARS" ; 

%JACKPV ( MATWGT , JKZONE, JKREP, 75, IDCNTRY BTDG01 , BSMMAT0 , 5, MERGED) ; 

PROC PRINT DATA = FINAL NOOBS ; 

VAR IDCNTRY BTDG01 N MATWGT MNPV MNPV_SE PCT PCT_SE ; 

FORMAT IDCNTRY COUNTRY. BTDG 01 BTDG. N 6.0 MATWGT 10.0 
MNPV MNPV_SE PCT PCT_SE 6.2 ; 


Exhibit 3.21 : Output for Example Teacher-level Analysis (EXAMPLE 5) 


IDCNTRY 

BTDG 01 

N 

MATWGT 

MNPV 

MNPV_SE 

PCT 

PCT_SE 

AUSTRALIA 

20 YEARS OR MORE 

2096 

68987 

519 . 11 

8 . 07 

36 . 54 

4 . 03 

AUSTRALIA 

AT LEAST 10 BUT LESS THAN 20 YEARS 

1395 

40605 

513 . 05 

10.77 

21 . 51 

3.39 

AUSTRALIA 

AT LEAST 5 BUT LESS THAN 10 YEARS 

888 

33462 

503 . 68 

17 . 14 

17.73 

3 . 18 

AUSTRALIA 

LESS THAN 5 YEARS 

1350 

45721 

485.34 

8.45 

24.22 

3.38 

BAHRAIN 

20 YEARS OR MORE 

966 

2317 

433.48 

7 . 02 

19.25 

2.20 

BAHRAIN 

AT LEAST 10 BUT LESS THAN 20 YEARS 

2295 

6492 

404 . 07 

3.72 

53 . 93 

3.56 

BAHRAIN 

AT LEAST 5 BUT LESS THAN 10 YEARS 

732 

2077 

403.36 

5 . 83 

17.25 

2 .68 

BAHRAIN 

LESS THAN 5 YEARS 

448 

1152 

429 . 50 

9 . 09 

9 . 57 

1.93 

ARMENIA 

20 YEARS OR MORE 

3508 

21506 

466 . 66 

3 . 90 

62 . 92 

3.71 

ARMENIA 

AT LEAST 10 BUT LESS THAN 20 YEARS 

1716 

10255 

464.48 

6 . 02 

30 . 00 

3.34 

ARMENIA 

AT LEAST 5 BUT LESS THAN 10 YEARS 

200 

1423 

473 . 15 

24 . 95 

4 . 16 

1.56 

ARMENIA 

LESS THAN 5 YEARS 

134 

999 

474.44 

18.40 

2 . 92 

1.40 

CHILE 

20 YEARS OR MORE 

2788 

116439 

415.30 

4.55 

49 . 15 

3 . 81 

CHILE 

AT LEAST 10 BUT LESS THAN 20 YEARS 

878 

35914 

416 . 01 

9.99 

15 . 16 

2 . 92 

CHILE 

AT LEAST 5 BUT LESS THAN 10 YEARS 

689 

31741 

421.22 

12 . 08 

13.40 

2 . 76 

CHILE 

LESS THAN 5 YEARS 

1166 

52819 

421.27 

6.30 

22.29 

3.36 
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3.9 TIMSS Analyses with School-level Variables 


Because TIMSS 2011 has representative samples of schools, it is possible to compute reasonable 
statistics with schools as units of analysis. However, the school samples were designed to optimize 
the student samples and the student-level estimates. For this reason, it is preferable to analyze 
school-level variables as attributes of students, rather than as elements in their own right. Therefore, 
analyzing school data should be done by linking the students to their schools. 

In general, to perform analyses using the school background data fdes, users should do the 
following: 

1 . Identify the variables of interest in the school and student background data files and note any 
specific national adaptations to the variables. 

2. Retrieve the relevant variables from the school background data files, including analysis 
variables, classification variables, identification variables (IDCNTRY and IDSCHOOL), and any 
other variables used in the selection of cases. 

3. Retrieve the relevant variables from the student background data files, including plausible values 
of achievement, classification variables, identification variables (IDCNTRY and IDSCHOOL), 
sampling (JKZONE and JKREP) and weighting (TOTWGT) variables, and any other variables 
used in the selection of cases. 

4. Merge the school background data files with the student background data files using the variables 
IDCNTRY and IDSCHOOL. 

5. Perform any necessary variable transformations or recodes. 

6. Use the macros JACKGEN and JACKREG, or JACKPV and JACKREGP if plausible values are 
involved, with the appropriate parameters. 

7. Specify the location of the data files (<datpath>) and the macros (<macpath>). 

8. Print the results file. 

Our example of an analysis using school background data will compute the percentages of eighth 
grade students, with their average mathematics achievement, who attended schools composed of 
students with different levels of economic background. The results of this analysis are presented in 
Exhibit 5.4 of TIMSS 2011 International Results in Mathematics and are replicated here in 
Exhibit 3.22. 

The information for this analysis is found in the school-level derived variable BCDG03, where 
schools are characterized as being composed of more affluent students than disadvantaged students, 
composed of more disadvantaged students than affluent students, or composed of neither more 
affluent students nor more disadvantaged students. 5 


5 


Supplement 3 to this User Guide describes the construction of the school-level derived variable BCDG03 from responses to questions posed to the 
school principals. 
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Exhibit 3.22: Exhibit of Example School-level Analysis, Taken from 

TIMSS 2011 International Results in Mathematics (Exhibit 5.4) 


Exhibit 5.4: School Composition by Student Economic Background 


TIMSS 2011 

Mathematics 


8 th 


Grade 


Reported by Principals 

More Affluent - Schools Where More More Disadvantaged - Schools Where 

than 25% of Students Come from .. _ _ More than 25% of Students Come 

- . „ .. ... Neither More Affluent nor „ . .. . 

Economically Affluent Homes and Not from Economically Disadvantaged 

Country More than 25% from Economically ° re ,Sa vanta 9 e Homes and Not More than 25% from 

Disadvantaged Homes Economically Affluent homes 


Percent 

Average 

Percent 

Average 

Percent 

Average 

of Students 

Achievement 

of Students 

Achievement 

of Students 

Achievement 


Armenia 

35 (3.7) 

484 (5.9) 

24 (3.6} 

461 (7.5) 

41 (3.7) 

455 (5.0) 

Australia 

32 (3.4) 

543 (112) 

39 (3.7} 

507 (6.1) 

29(3.1) 

476 (7.5) 

Bahrain 

45 (0.3) 

420 (3.2) 

28 (0.2) 

408 (2.7) 

27(0.3) 

395 (3.7) 

Chile r 

12 (2.3) 

474 (13.0) 

32 (4.1) 

439 (6.0) 

56(3.9) 

399 (4.8) 

Chinese Taipei 

17 (2.7) 

649 (7.9) 

69 (3.8) 

604 (4.2) 

14(2.9) 

577 (13.5) 

England 

28 (4.1) 

553 (11.0) 

50 (4.5) 

498 (8.9) 

22 (4.3) 

487 (10.9) 

Finland r 

30 (3.4) 

519 (4.0) 

67 (3.8) 

513 (3.0) 

3 (15) 

486 (3.5) 

Georgia 

11 (2.0) 

436 (13.7) 

44 (4.4) 

438 (6.8) 

45 (4.2) 

417 (6.8) 

Ghana 

7 (2.0) 

392 (13.9) 

18 (3.4) 

331 (10.6) 

75 (3.6) 

321 (5.2) 

Hong Kong SAR 

11 (3.0) 

628 (11.8) 

37(5.1) 

609 (10.2) 

53 (4.8) 

561 (7.8) 

Hungary 

16 (2.7) 

535 (7.4) 

33 (4.1) 

531 (4.9) 

50 (4.3) 

478 (5.6) 

Indonesia 

16 (3.3) 

426 (9.9) 

28 (4.6) 

400 (8.1) 

56(4.6) 

369 (6.0) 

Iran, Islamic Rep. of 

20 (2.7) 

472 (112) 

25 (3.5) 

429 (9.1) 

54(3.8) 

390 (5.2) 

Israel 

28 (3.5) 

556 (7.8) 

30 (4.5) 

526 (8.8) 

42 (3.9) 

481 (8.8) 

Italy 

40 (3.7) 

515 (3.7) 

47 (3.9) 

495 (3.8) 

13 (2.6) 

465 (8.9) 

Japan 

46 (4.4) 

582 (4.5) 

44 (4.5) 

564 (4.1) 

10 (2.9) 

548 (9.0) 

Jordan r 

32 (3.5) 

431 (7.0) 

25 (2.9) 

402 (9.7) 

43 (3.9) 

388 (6.3) 

Kazakhstan 

75 (3.5) 

487 (4.4) 

20 (3.4) 

493 (11.0) 

5 (1.8) 

462 (22.5) 

Korea, Rep. of 

18 (3.3) 

653 (5.8) 

51 (4.3) 

612 (2.6) 

32 (3.9) 

591 (4.6) 

Lebanon r 

21 (4.1) 

491 (8.8) 

34 (4.2) 

455 (8.7) 

45 (5.0) 

435 (5.3) 

Lithuania 

23 (3.6) 

537 (6.5) 

39 (4.4) 

499 (4.3) 

38 (4.0) 

487 (4.5) 

Macedonia, Rep. of r 

38 (3.6) 

458 (7.9) 

30 (4.1) 

428 (10.0) 

32 (3.9) 

401 (9.7) 

Malaysia 

26 (3.2) 

467 (10.5) 

23 (3.3) 

452 (12.4) 

52 (4.1) 

424 (8.8) 

Morocco r 

6 (1.4) 

422 (15.0) 

13 (2.5) 

393 (9.8) 

81 (2.9) 

361 (2.6) 

New Zealand 

30 (5.6) 

522 (6.9) 

47 (5.8) 

485 (7.4) 

24 (4.0) 

450 (10.6) 

Norway 

-- 


-- 

-- 

-- 


Oman 

43 (3.1) 

386 (4.6) 

26 (2.6) 

360 (5.6) 

31 (3.1) 

339 (5.8) 

Palestinian Nat'l Auth. 

44 (4.2) 

411 (6.5) 

23 (3.9) 

402 (8.7) 

33(3.7) 

393 (6.1) 

Qatar r 

81 (0.2) 

403 (4.3) 

16 (0.2) 

448 (6.6) 

3(0.1) 

435 (18.2) 

Romania 

18 (2.9) 

479 (12.7) 

29 (4.2) 

471 (8.1) 

52 (4.3) 

447 (6.2) 

Russian Federation 

58 (3.5) 

553 (5.1) 

25 (2.8) 

527 (4.4) 

16(3.1) 

513 (10.3) 

Saudi Arabia r 

40 (4.4) 

405 (7.5) 

30 (4.4) 

394 (10.5) 

29 (4.1) 

382 (8.2) 

Singapore 

27 (0.0) 

643 (5.9) 

61 (0.0) 

604 (4.9) 

11 (0.0) 

569(11.6) 

Slovenia 

40 (3.8) 

510 (4.4) 

45 (4J) 

506 (2.7) 

15 (2.7) 

489 (6.8) 

Sweden r 

74 (4.4) 

490 (2.6) 

21 (4.1) 

472 (5.5) 

5 (1.8) 

466(11.9) 

Syrian Arab Republic r 

37 (4.2) 

388 (8.0) 

27 (4.3) 

392 (9.5) 

36 (4.4) 

371 (8.2) 

Thailand 

20 (3.0) 

466 (13.9) 

24 (3.6) 

437 (9.5) 

57 (4.4) 

410 (5.7) 

Tunisia 

23 (3.3) 

439 (9.6) 

29(3.3) 

432 (3.9) 

48(321) 

411 (3.0) 

Turkey 

17 (2.6) 

533 (11.6) 

25 (3.3) 

455 (6.0) 

59 (3.8) 

428 (5.1) 

Ukraine 

13 (2.7) 

486 (14.1) 

29 (3.9) 

486 (7.4) 

59 (4.5) 

472 (5.1) 

United Arab Emirates r 

70 (2.0) 

459 (3.4) 

17 (1.9) 

442 (7.3) 

13 (1.4) 

441 (5.6) 

United States 

22 (1.9) 

543 (5.8) 

23 (1.9) 

526 (6.1) 

55 (1.9) 

490 (3.4) 

International Avg. 

32 (0.5) 

494 (1.4) 

33 (0.6) 

471 (1.2) 

36(0.5) 

448(13) 


Ninth Grade Participants 


Botswana 

13 (3.0) 

432 (10.9) 

24 (4.0) 

401 (4.4) 

63 (4.6) 

384 (2.7) 

Honduras s 

5 (1.6) 

383 (12.5) 

14 (3.4) 

358 (12.3) 

82 (3.6) 

333 (4.4) 

South Africa r 

8 (1.3) 

487 (14.4) 

12 (2.6) 

356 (15.0) 

80 (2.7) 

339 (3.2) 

Benchmarking Participants 

Alberta, Canada 

39 (4.1) 

517 (3.6) 

43 (4.8) 

505 (3.3) 

18 (3.8) 

482 (5.9) 

Ontario, Canada 

37 (4.1) 

523 (5.1) 

36 (4.7) 

510 (3.8) 

27 (4.5) 

498 (5.2) 

Quebec, Canada r 

51 (4.1) 

542 (4.3) 

32 (3.8) 

523 (5.2) 

17(3.5) 

514 (6.3) 

Abu Dhabi, UAE r 

76 (4.1) 

453 (6.1) 

17 (3.6) 

429 (10.3) 

7 (2.4) 

446 (14.9) 

Dubai, UAE r 

71 (0.3) 

484 (3.2) 

12 (0.2) 

449 (2.9) 

16 (0.2) 

434 (3.8) 

Alabama, US r 

17 (4.4) 

492 (19.0) 

5 (3.4) 

481 (41.0) 

78 (5.6) 

455 (6.1) 

California, US r 

16 (4.2) 

541 (12.3) 

20 (5.2) 

532 (16.7) 

64 (5.4) 

467 (5.8) 

Colorado, US r 

21 (5.7) 

525 (9.1) 

34 (6.6) 

526 (10.9) 

46 (7.4) 

500 (12.5) 

Connecticut, US r 

43 (6.1) 

565 (7.8) 

27 (6.1) 

528 (10.3) 

30 (5.9) 

455 (8.6) 

Florida, US r 

6 (3.4) 

500 (18.4) 

37 (5.6) 

535 (11.1) 

58 (6.0) 

499 (8.8) 

Indiana, US r 

13 (4.5) 

573 (7.5) 

29 (5.3) 

524 (10.1) 

58 (5.9) 

509 (6.6) 

Massachusetts, US 

29 (6.8) 

589 (9.1) 

45 (6.6) 

562 (8.0) 

26 (4.2) 

521 (13.4) 

Minnesota, US 

18 (3.2) 

583 (16.6) 

45 (7.1) 

546 (5.4) 

37 (7.6) 

530 (8.4) 

North Carolina, US r 

14 (5.6) 

560 (16.1) 

23 (6.4) 

551 (10.9) 

63 (6.7) 

519 (10.5) 


( ) Standard errors appear In parentheses. Because of rounding some results may appear inconsistent. 
A dash (-) indicates comparable data not available. 


£ 

5 

9 


An "r" Indicates data are available for at least 70% but less than 85% of the students. An "s” Indicates data are available for at least 50% but less than 70% of the students. 
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Because we are using a school-level variable, we need to use the school background data fdes 
and the student background data files to find the variables. From the school background data files, 
we need the variable that contains the information on the schools’ composition of students based on 
their economic background (BCDG03) and the identification variables IDCNTRY and IDSCHOOL 
that will allow us to link the school data to the student data. 

Next, we retrieve the variables of interest from the student background data files. We need the 
country and school identification variables (IDCNTRY and IDSCHOOL) necessary to merge the 
school data to the student data. We also need the jackknife replication variables (JKZONE and 
JKREP), the student weighting variable (TOTWGT), and the eighth grade mathematics achievement 
plausible values (BSMMAT01 through BSMMAT05). 

We then proceed to merge the school data with the student data using the variables IDCNTRY 
and IDSCHOOL and use the macro JACKPV to obtain the percentages of students, with their 
average mathematics achievement, in each category of the variable BCDG03 for each country. For 
this analysis, we will use the data for all available countries, making use of an aggregated school file 
BCGALLM5 and an aggregated student file BSGALLM5. These aggregated files were created with 
the JOIN macro. 

The SAS program that implements this example is presented in Exhibit 3.23 and is included in 
the International Database under the name EXAMPLE6.SAS. The results of this program are 
displayed in Exhibit 3.24, edited to show only the first four countries for the sake of brevity. Note 
that one of the steps in this program is to select only those students who have non-missing data in our 
variable of interest BCDG03. 

In Exhibit 3.24, each country’s results are presented on three lines, one for each value of the 
BCDG03 variable. The results are presented much in the same manner as in previous examples, 
where the first column identifies the countries and the second column describes the category of 
BCDG03 being reported. 

As shown in the first three lines of results, 32.49 percent of eighth grade students in Australia 
attended schools with more affluent students than disadvantaged students (standard error of 3.36), 
38.54 percent attended schools with neither more affluent students nor more disadvantaged students 
(standard error of 3.74), and 28.97 percent attended schools with more disadvantaged students than 
affluent students (standard error of 3.1 1). Also, the estimated average mathematics achievement 
was 542.55 (standard error of 1 1.18) for eighth grade students in schools with more affluent students, 
507.36 (standard error of 6.08) for students in schools with neither more affluent students nor more 
disadvantaged students, and it was 475.74 (standard error of 7.50) for students in schools with more 
disadvantaged students. 
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Exhibit 3.23: Example SAS Program for School-level Analysis (EXAMPLE6.SAS) 


LIBNAME Til "<datpath>" ; 

% INCLUDE " <macpath> JACKPV . SAS " ; 

PROC SORT DATA = Til . BCGALLM5 OUT = BCGALLM5 ; 
BY IDCNTRY IDSCHOOL ; 

PROC SORT DATA = Til . BSGALLM5 OUT = BSGALLM5 ; 
BY IDCNTRY IDSCHOOL ; 

DATA MERGED ; 

MERGE BCGALLM5 (IN = INBCG) 

BSGALLM5 (IN = INBSG) ; 

BY IDCNTRY IDSCHOOL ; 

IF INBCG AND INBSG ; 

DATA MERGED ; 

SET MERGED ; 

WHERE NMISS (BCDG03 ) = 0 ; 

PROC FORMAT LIBRARY = WORK ; 


VALUE COUNTRY 

< list 2011 country formats > 

VALUE BCDG 

1 = "MORE AFFLUENT LESS DISADVANTAGED" 

2 = "NEITHER" 

3 = "MORE DISADVANTAGED LESS AFFLUENT" ; 

%JACKPV ( TOTWGT , JKZONE, JKREP, 75, IDCNTRY BCDG03 , BSMMATO , 5, MERGED) ; 

PROC PRINT DATA = FINAL NOOBS ; 

VAR IDCNTRY BCDG03 N TOTWGT MNPV MNPV_SE PCT PCT_SE ; 

FORMAT IDCNTRY COUNTRY. BCDG03 BCDG. N 6.0 TOTWGT 10.0 
MNPV MNPV_SE PCT PCT_SE 6.2 ; 


Exhibit 3.24: Output for Example School-level Analysis (EXAMPLE 6) 


IDCNTRY 

BCDG03 


N 

TOTWGT 

MNPV 

MNPV_SE 

PCT 

PCT_SE 

AUSTRALIA 

MORE AFFLUENT LESS 

DISADVANTAGED 

2118 

71105 

542.55 

11 . 18 

32.49 

3.36 

AUSTRALIA 

NEITHER 


2535 

84358 

507.36 

6 . 08 

38 . 54 

3 . 74 

AUSTRALIA 

MORE DISADVANTAGED 

LESS AFFLUENT 

1800 

63400 

475.74 

7.50 

28 . 97 

3.11 

BAHRAIN 

MORE AFFLUENT LESS 

DISADVANTAGED 

1954 

5157 

419 . 65 

3.19 

45.30 

0.32 

BAHRAIN 

NEITHER 


1143 

3173 

407.57 

2.72 

27 . 87 

0.23 

BAHRAIN 

MORE DISADVANTAGED 

LESS AFFLUENT 

1051 

3055 

395.48 

3.69 

26 . 83 

0.34 

ARMENIA 

MORE AFFLUENT LESS 

DISADVANTAGED 

2085 

12047 

483.52 

5 . 89 

34 . 78 

3 . 70 

ARMENIA 

NEITHER 


1329 

8398 

460 . 52 

7.53 

24.25 

3.59 

ARMENIA 

MORE DISADVANTAGED 

LESS AFFLUENT 

2158 

14191 

455.43 

5 . 03 

40 . 97 

3 .68 

CHILE 

MORE AFFLUENT LESS 

DISADVANTAGED 

811 

22830 

474.35 

13 . 02 

12 . 16 

2.32 

CHILE 

NEITHER 


1391 

59452 

439.36 

5 . 98 

31.66 

4 . 07 

CHILE 

MORE DISADVANTAGED 

LESS AFFLUENT 

2119 

105493 

398 . 81 

4.76 

56 . 18 

3 . 86 
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Chapter 4 

The TIMSS 201 1 
International Database Files 

4.1 Overview 

The TIMSS 2011 International Database contains achievement data and student, home, teacher, and 
school background data collected in the 63 countries and 14 benchmarking participants that took part 
in TIMSS 201 1. 1 Exhibit 4.1 lists all of the TIMSS 2011 countries and benchmarking participants, 
along with identifying codes used in the International Database. The database also contains materials 
that provide additional information on its structure and contents. This chapter describes the contents 
of the database with a special emphasis on the actual data files used in Chapters 2 and 3 of this User 
Guide to analyze the TIMSS 2011 assessment results. The next section introduces this User Guide 
and its supplements. The following section describes the item-related materials included in the 
international database. The final five sections present the different data files and related materials 
included in the database, as described in Exhibit 1 . 1 of Chapter 1 . 

4.2 The TIMSS 2011 User Guide 

The TIMSS 2011 User Guide is an important resource to the TIMSS 2011 International Database 
because it provides a detailed description of its structure and contents, as well as instructions to make 
the best use of its data. The User Guide also includes a series of four supplements in printable PDF 
format: 

• Supplement 1 : International Version of the TIMSS 2011 Background and Curriculum 
Questionnaires — Supplement 1 includes the international version of all background and 
curriculum questionnaires administered in TIMSS 201 1. It is a good reference guide to 
understand what questions were asked and the variable names under which the responses are 
recorded in the International Database. 

• Supplement 2: National Adaptations of International Background Questionnaires — Supplement 2 
provides details on all national adaptations that were applied to the national version of all 
TIMSS 2011 background questionnaires. Users should refer to this supplement for any special 
national adaptations to background variables that could potentially affect the results of analyses. 


The TIMSS 201 1 International Database also includes data from the PIRLS 2011 “Learning to Read” survey (a home questionnaire administered to 
parents) for countries that administered the TIMSS and PIRLS 201 1 fourth grade assessments to the same sample of students. 
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Exhibit 4.1 : Countries Participating in TIMSS 2011 


Country 

ISO Code 

Assessment 

Alpha 

Numeric 

Fourth Grade 

Eighth Grade 

Armenia 

ARM 

051 

• 

• 

Australia 

AUS 

036 

• 

• 

Austria 

AUT 

040 

• 


Azerbaijan 

AZE 

031 

• 


Bahrain 

BHR 

048 

• 

• 

Belgium (Flemish) 

BFL 

956 

• 


Chile 

CHL 

152 

• 

• 

Chinese Taipei 

TWN 

158 

• 

• 

Croatia 

HRV 

191 

• 


Czech Republic 

CZE 

203 

• 


Denmark 

DNK 

208 

• 


England 

ENG 

926 

• 

• 

Finland 

FIN 

246 

• 

• 

Georgia 

GEO 

268 

• 

• 

Germany 

DEU 

2Z6 

• 


Ghana 

GHA 

288 


• 

Hong Kong SAR 

HKG 

344 

• 

• 

Hungary 

HUN 

348 

• 

• 

Indonesia 

IDN 

360 


• 

Iran, Islamic Rep. of 

IRN 

364 

• 

• 

Ireland 

IRL 

3Z2 

• 


Israel 

ISR 

3Z6 


• 

Italy 

ITA 

380 

• 

• 

Japan 

JPN 

392 

• 

• 

Jordan 

JOR 

400 


• 

Kazakhstan 

KAZ 

398 

• 

• 

Korea, Rep. of 

KOR 

410 

• 

• 

Kuwait 

KWT 

414 

• 


Lebanon 

LBN 

422 


• 

Lithuania 

LTU 

440 

• 

• 

Macedonia, Rep. of 

MKD 

807 


• 

Malaysia 

MYS 

458 


• 

Malta 

MLT 

470 

• 


Morocco 

MAR 

504 

• 

• 

Netherlands 

NLD 

528 

• 


New Zealand 

NZL 

554 

• 

• 

Northern Ireland 

NIR 

928 

• 


Norway 

NOR 

578 

• 

• 

Oman 

OMN 

512 

• 

• 

Palestinian Nat'l Auth. 

PSE 

275 


• 


TIMSS 201 1 USER GUIDE FOR THE INTERNATIONAL DATABASE 

78 CHAPTER 4 



TIMSS & PIRLS 

International Study Center 

Lynch School of Education, Boston College 



Exhibit 4.1 : Countries Participating in TIMSS 2011 (Continued) 


ISO Code 

Assessment 

country 

Alpha 

Numeric 

Fourth Grade 

Eighth Grade 

Poland 

POL 

616 

• 


Portugal 

PRT 

620 

• 


Qatar 

QAT 

634 

• 

• 

Romania 

ROM 

642 

• 

• 

Russian Federation 

RUS 

643 

• 

• 

Saudi Arabia 

SAU 

682 

• 

• 

Serbia 

SRB 

688 

• 


Singapore 

SGP 

702 

• 

• 

Slovak Republic 

SVK 

703 

• 


Slovenia 

SVN 

705 

• 

• 

Spain 

ESP 

724 

• 


Sweden 

SWE 

752 

• 

• 

Syrian Arab Republic 

SYR 

760 


• 

Thailand 

THA 

764 

• 

• 

Tunisia 

TUN 

788 

• 

• 

Turkey 

TUR 

792 

• 

• 

Ukraine 

UKR 

804 


• 

United Arab Emirates 

ARE 

784 

• 

• 

United States 

USA 

840 

• 

• 

Yemen 

YEM 

887 

• 


Out-of-grade Countries 

Botswana (6 th and 9 th ) 

BWA 

072 

• 

• 

Honduras (6 th and 9 th ) 

HND 

340 

• 

• 

South Africa (9 th ) 

ZAF 

710 


• 

Yemen (6 th ) 

YE6 

6887 

• 


Benchmarking Participants 

Alberta, Canada 

CAB 

9134 

• 

• 

Ontario, Canada 

COT 

9132 

• 

• 

Quebec, Canada 

CQU 

9133 

• 

• 

Abu Dhabi, UAE 

AAD 

7842 

• 

• 

Dubai, UAE 

ADU 

7841 

• 

• 

Alabama, US 

UAL 

10400 


• 

California, US 

UCA 

11100 


• 

Colorado, US 

UCO 

10800 


• 

Connecticut, US 

UCT 

10900 


• 

Florida, US 

UFL 

11200 

• 

• 

Indiana, US 

UIN 

11800 


• 

Massachusetts, US 

UMA 

12500 


• 

Minnesota, US 

UMN 

12700 


• 

North Carolina, US 

UNC 

13700 

• 

• 
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• Supplement 3: Variables Derived from the Student, Home, Teacher, and School Questionnaire 
Data — Supplement 3 describes how the derived background variables were computed to produce 
exhibits in the TIMSS 2011 International Reports (Mullis, Martin, Foy, & Arora, 2012; and 
Martin, Mullis, Foy, & Stanco, 2012). 

• Supplement 4: TIMSS 2011 Sampling Stratification Information — Supplement 4 provides the 
labels assigned to the national explicit and implicit strata defined during the sampling process. 

4.3 TIMSS 2011 Achievement Items Documentation 

A number of documents related to the TIMSS 2011 achievement items are available for download 
along with the International Database. They include detailed information on the TIMSS 2011 
released items, summary information on the TIMSS 2011 items, as well as the IRT item parameters 
estimated from the TIMSS 2011 concurrent item calibration. 

The International Database includes PDF documents of the TIMSS 2011 released mathematics 
and science achievement items at the fourth and eighth grades. The documents include the items 
themselves with descriptive in formation and the scoring guides for the constructed response items. 
The International Database also includes percent correct statistics for all released items in Excel and 
PDF format. There are separate files by grade and subject. 

Achievement item information files are provided to enable users of the TIMSS 2011 database to 
readily produce summaries of item characteristics. There are separate achievement item information 
files for the fourth and eighth grades, with separate tables by subject within. These files are available 
in Excel format and include the following information for each item in the TIMSS 2011 assessments: 

• The item’s permanent and unique identifier; 

• The item’s block and its sequential location within the block; 

• The item’s label; 

• The item’s content domain and cognitive domain; 

• The item’s type, either multiple choice or constructed response; 

• The number of options for a multiple choice item; 

• The correct response key for a multiple choice item; 

• The item’s point value; 

• An indicator showing if the item was included in the IRT scaling; and 

• An indicator showing if the item was released after the 2011 assessment. 

The International Database also includes Excel files with the IRT item parameters estimated for 
all TIMSS 2007 and TIMSS 2011 items from the concurrent item calibrations. There is an Excel file 
for each grade that contains separate tables for overall mathematics and overall science. These same 
item parameters are presented in the scaling section of Methods and Procedures in TIMSS and 
PIRLS 2011 (Martin & Mullis, 2012). Finally, the database also includes Excel tables with the scale 
transformations that were applied to the TIMSS 2011 mathematics and science scales. 
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4.4 The TIMSS 201 1 Data Files 


The TIMSS 2011 International Database includes the actual data from all instruments administered 
to the students, their parents, their teachers, and their school principals. This includes the student 
responses to the achievement items and the responses to the student, home, teacher, and school 
background questionnaires. These data files also include the achievement scores estimated for 
participating students as well as background variables derived for reporting in the TIMSS 2011 
International Reports. National Research Coordinators’ responses to the curriculum questionnaires 
also are part of the International Database and are described later in this chapter. 

This section describes the contents and format of the TIMSS 2011 data files. With the exception 
of the curriculum data files, the TIMSS data files are provided in SAS (SAS Institute, 2011) export 
format (.EXP) and SPSS (IBM Corporation, 2012) format (.SAV). Data files are provided for each 
country that participated in TIMSS 2011 and for which internationally comparable data are available. 
The file names given to the various data file types are shown in Exhibit 4.2. For example, 
ASGNORM5.SAV is an SPSS file that contains Norway’s TIMSS 2011 fourth grade student 
background data. For each file type, a separate data file is provided for each participating country. 

All data files and the variables they contain are described in the following sections. 

Exhibit 4.2: TIMSS 201 1 Data File Names 


File Names 

Descriptions 

ACG»»»M5 

Fourth grade school background data files 

ASA*»»M5 

Fourth grade student achievement data files 

ASG»««M5 

Fourth grade student background data files 

ASH*»»M5 

Fourth grade home background data files 

ASR*»»M5 

Fourth grade within-country scoring reliability data files 

AST#«*M5 

Fourth grade student-teacher linkage files 

ATG*»»M5 

Fourth grade teacher background data files 

BCG»««M5 

Eighth grade school background data files 

BSA*«»M5 

Eighth grade student achievement data files 

BSG»««M5 

Eighth grade student background data files 

BSR*»»M5 

Eighth grade within-country scoring reliability data files 

BST«**M5 

Eighth grade student-teacher linkage files 

BTM»»*M5 

Eighth grade mathematics teacher background data files 

BTS«**M5 

Eighth grade science teacher background data files 


• •• = 3-character country abbreviation based on the ISO 3166 coding scheme 
(see Exhibit 4.1). 
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4.4.1 TIMSS 201 1 Student Achievement Data Files (ASA/BSA) 


The TIMSS 2011 student achievement data files contain the student responses to the individual 
achievement items in the TIMSS 2011 assessments. The student achievement data files are best 
suited for performing item-level analyses. Achievement scores (plausible values) for all of the 
TIMSS 2011 achievement scales are available in the student achievement data files, as well as in the 
student background data files and student-teacher linkage data files. 

Students who participated in TIMSS 2011 were administered one of 14 assessment booklets, 
each with a series of mathematics and science items. Some of these items were multiple choice 
items and some were constructed response items. The student achievement data files contain the 
actual responses to the multiple choice questions and the codes assigned to the constructed response 
items through the scoring guides. 

Item Variable Naming Convention 

The achievement item variable names are based on an 8-character alphanumeric code 
(e.g., M051064A), which adheres to the following rules: 

• The first character is either “M” for mathematics items or “S” for science items. 

• The second and third characters indicate the assessment cycle when the item was first used in 
TIMSS. The code “01” was used for items introduced in TIMSS 1995. The items in the 
TIMSS 2011 assessment have either “03” for items produced in 2003, “04” for items produced in 
2007, or “05” for new items in 201 1. 

• The fourth character is either “1” for fourth grade items, or “2” for eighth grade items. 

• The fifth through seventh characters are a three-digit number used to uniquely identify the items. 

• The eighth character indicates the item part, and appears only when required. It is generally a 
letter from “A” to “F,” depending how many parts there are to a particular item. The letter “Z” is 
used to represent a derived item where the scores of its item parts are combined into a single 
derived item. 

For example, M05 1064A is the first part of a fourth grade mathematics item produced in 201 1 
whose unique sequence number is 064. 

Item Response Code Values 

A series of conventions also were adopted to code the data included in the TIMSS data files. 

This section describes these conventions for the achievement items. 

The values assigned to each of the achievement item variables depend on the item format. For 
multiple choice items, numerical values from 1 through 4 were used to correspond to the response 
options A through D, respectively. For these items, the correct response code is included as part of 
the variable label in the achievement codebook file, and SAS and SPSS programs are included as 
part of the International Database to score these items. 


The TIMSS 201 1 booklet design is described in Chapter 4 of the TIMSS 2011 Assessment Frameworks (Mullis, Martin, Ruddock, O'Sullivan, 
& Preuschoff, 2009). 

3 The SPSS version of these programs function much like their SAS counterparts, described in Section 3.4 of Chapter 3. 
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Each constructed response item had its own scoring guide that relied on a two-digit scoring 
scheme to provide diagnostic information. 4 The first digit designated the correctness level of the 
response: 2 for a two-point response, 1 for a one-point response, and 7 for an incorrect response. The 
second digit, combined with the first, represented a diagnostic code used to identify specific types of 
approaches, strategies, or common errors and misconceptions in responding to the item. A second 
digit of 0 through 5 was used for pre-defined international codes at each correctness level, while a 
second digit of 9 corresponded to “other” types of responses which fell within the appropriate 
correctness level but which did not fit any of the pre-defined international codes. A special code, 99, 
was used for completely blank responses. 

For some items, students were asked to provide more than one answer, each one being scored 
separately. The pattern of responses across these item parts determined the score on the item as a 
whole. These multi-part items fell into one of two categories. In the first instance, students provided 
two answers and the score on the item was the number of correct responses. In the second instance, 
students provided more than one answer and the item received a score of one if all responses were 
correct. The total score for the item is contained in a derived variable, indicated by a final character 
of Z in its item variable name. For example, derived variable S05 1 188Z contains the combined score 
for its five parts labeled S05 1 188A through S05 1 188E. For all of these items, each item part was 
worth one point, and the derived items were given the code values shown in Exhibit 4.3. 


Exhibit 4.3: 

TIMSS 201 1 Derived Item Variable Codes 

Code 

Description 

For 2-point Derived Items: 

20 

Full credit (1 point on both part A and part B) 

10 

Partial credit (1 point on either part A or part B) 

70 

No credit (no points on both part A and part B) 

For 1-point Derived Items: 

10 

Full credit (all parts correct) 

70 

No credit (at least one part incorrect) 


Codes for Missing Values 

A subset of the values for each variable type was reserved for specific codes related to different 
categories of missing data. We recommend that the user read the following section with particular 
care, because the way in which these missing codes are used may have major consequences for 
analyses. 


4 Scoring guides for the released items are provided in the Released Items document, available for download at 
http://timssandpirls. be. edu/timss201 1 /international-database, html. 
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Omitted Response Codes (SAS: . ; SPSS: 9, 99) 


“Omitted” response codes were used for items that a student should have answered but did not. 

An omitted response code was given when an item was left blank or when two or more response 
options were checked for a multiple choice item. 

Not Administered Response Codes (SAS: .A ; SPSS: sysmis) 

Special codes were given to items that were “Not Administered” to distinguish these cases from data 
that were missing due to non-response. In general, the not administered code was used when an item 
was not administered, either by design arising from the rotation of items across the assessment 
booklets, or unintentionally when an item was misprinted or otherwise unavailable for a student to 
respond. The not administered code was used in the following cases: 

• Achievement item not assigned to the student — All students participating in TIMSS received 
only one of the 14 test booklets. All variables corresponding to items that were not present in a 
student’s assigned booklet were coded as “Not Administered.” 

• Student absent from session — When a student was not present for a particular testing session, 
all variables relevant to that session were coded as “Not Administered.” 

• Item left out or misprinted — When a particular item (or a whole page) was misprinted or 
otherwise not available to the student, the corresponding variable was coded as “Not 
Administered.” 

• Item deleted or mistranslated — An item identified during translation verification or item review 
as having a translation error such that the nature of the question was altered, or as having poor 
psychometric properties, was coded as “Not Administered.” 

Not Reached Response Codes (SAS: .R ; SPSS: 6, 96) 

An item was considered not reached when, within part 1 or part 2 of a booklet, the item itself and the 
item immediately preceding it were not answered, and there were no other items completed in the 
remainder of that part of the booklet. For most purposes, TIMSS 2011 treated the not-reached items 
as incorrect responses, except during the item calibration step of the IRT scaling, when not-reached 
items were considered to have not been administered (see the scaling section in Methods and 
Procedures in TIMSS and PIRLS 2011). 

TIMSS Achievement Scores 

Achievement scales were produced for mathematics and science and their content and cognitive 
domains at both grades, as shown in Exhibit 4.4. A total of 14 achievement scales were produced at 
the fourth grade and 16 at the eighth grade. A detailed description of the TIMSS 2011 scaling 
approach and how these achievement scales were created is available in the scaling section of 
Methods and Procedures in TIMSS and PIRLS 2011. For each achievement scale, the TIMSS 2011 
database provides five separate estimates of each student’s score on that scale. The five estimated 
scores are kn own as “plausible values,” and the variability between them encapsulates the 
uncertainty inherent in the scale estimation process. 
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Exhibit 4.4: TIMSS 201 1 Achievement Scales at Fourth and Eighth Grades 


TIMSS 2011 Achievement Scales 


Overall 

MAT 

Mathematics 

SCI 

Science 



NUM 

Number 

LIF 

Life Science 

Fourth 

Content 

Domains 

GEO 

Geometric Shapes 
and Measurement 

PHY 

Physical Science 

Grade 


DAT 

Data Display 

EAR 

Earth Science 



KNO 

Knowing 

KNO 

Knowing 


Cognitive 

Domains 

APP 

Applying 

APP 

Applying 


REA 

Reasoning 

REA 

Reasoning 


Overall 

MAT 

Mathematics 

SCI 

Science 



NUM 

Number 

BIO 

Biology 


Content 

ALG 

Algebra 

CHE 

Chemistry 

Eighth 

Domains 

GEO 

Geometry 

PHY 

Physics 

Grade 


DAT 

Data and Chance 

EAR 

Earth Science 



KNO 

Knowing 

KNO 

Knowing 


Cognitive 

Domains 

APP 

Applying 

APP 

Applying 


REA 

Reasoning 

REA 

Reasoning 


The plausible values for any given scale are the best available measures of student achievement 
on that scale in the TIMSS 2011 International Database, and should be used as the outcome measure 
in any study of student achievement. Plausible values can be readily analyzed using the IEA 
International Database (IDB) Analyzer and the SAS programs described in this User Guide. 

The achievement score variable names are based on an eight-character alphanumeric code (e.g., 
ASMKNOOl), which adheres to the following rules: 

• The first character is either “A,” for a fourth grade score, or “B,” for an eighth grade score; 

• The second character is always “S,” to indicate it is a student score variable; 

• The third character is either “M,” for a mathematics score, or “S,” for a science score, whether it 
is an overall score or a content domain or cognitive domain score; 

• The fourth through sixth characters are a three-character code describing the achievement scale, 
as shown in Exhibit 4.4; and 

• The seventh and eighth characters are a two-digit number indicating the plausible value — “01,” 
“02,” “03,” “04,” or “05.” 

For example, ASMKNOOl is the first plausible value on the fourth grade mathematics knowing 
cognitive domain achievement scale. 
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TIMSS International Benchmarks of Achievement 

To help users of the TIMSS 2011 achievement results understand what performance on the overall 
mathematics and science achievement scales signifies in terms of the mathematics and science 
students know and can do, TIMSS identified four points on the overall mathematics and science 
scales to serve as International Benchmarks. As shown in Exhibit 4.5, the TIMSS International 
Benchmark scores are 625, 550, 475, and 400, which correspond to the Advanced International 
Benchmark, the High International Benchmark, the Intermediate International Benchmark, and the 
Low International Benchmark, respectively. TIMSS used a technique kn own as scale anchoring to 
summarize and describe student achievement at these four points on the scaled The TIMSS 2011 
International Reports present the results of this scale anchoring, and report the percentage of 
students in each country reaching each of the TIMSS International Benchmarks. 

Exhibit 4.5: TIMSS 201 1 International Benchmarks of Mathematics and Science Achievement 


Scale Score 

International Benchmark 

625 

Advanced International Benchmark 

550 

High International Benchmark 

475 

Intermediate International Benchmark 

400 

Low International Benchmark 


To assist analysts in using the TIMSS International Benchmarks in secondary analyses, the 
TIMSS 2011 International Database contains a set of variables indicating which International 
Benchmark the students reached. There is a benchmark variable for each plausible value of the 
overall mathematics and overall science scales at both grades. The International Benchmark variables 
follow the achievement score variable naming convention where the fourth through sixth positions 
have the letters “IBM.” Thus, ASMIBM01-05 are the five benchmark variables for fourth grade 
overall mathematics, ASSIBM01-05 the five benchmark variables for fourth grade overall science, 
BSMIBM01-05 for eighth grade overall mathematics, and BSSIBM01-05 for eighth grade overall 
science. The codes used for all the benchmark variables are described in Exhibit 4.6. 


The scale anchoring procedure is described in the scale anchoring section of Methods and Procedures in TIMSS and PIRLS 2011. 
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Exhibit 4.6: TIMSS 201 1 International Benchmark Variable Codes 


Code 

Description 

1 

Student performed below the Low International Benchmark 

2 

Student performed at or above the Low International Benchmark, but 
below the Intermediate International Benchmark 

3 

Student performed at or above the Intermediate International 
Benchmark, but below the High International Benchmark 

4 

Student performed at or above the High International Benchmark, but 
below the Advanced International Benchmark 

5 

Student performed at or above the Advanced International Benchmark 


4.4.2 TIMSS 201 1 Within-country Scoring Reliability Data Files (ASR/BSR) 

The TIMSS 2011 within-country scoring reliability data files contain data that can be used to 
investigate the reliability of the TIMSS constructed response item scoring. The scoring reliability 
data files contain one record for each assessment booklet that was double scored during the within- 
country scoring reliability exercise (see Operations and Quality Assurance in Methods and 
Procedures in TIMSS and PIRLS 2011). For each constructed response item in the 2011 assessment, 
the following three variables are included in the scoring reliability data files: 

• Original Score (two-digit score assigned by the first scorer); 

• Second Score (two-digit score assigned by the second scorer); and 

• Score Agreement (degree of agreement between the two scorers). 

It should be noted that the second score data were used only to evaluate within-country scoring 
reliability and were not used in computing the achievement scores included in the International 
Database and presented in the TIMSS 2011 International Reports. 

Scoring Reliability Variable Naming Convention 

The variable names for the Original Score, Second Score, and Score Agreement variables are based 
on the same naming convention as that for the achievement item variables discussed above. 

The second character in the variable name differentiates between the three reliability variables: 

• The Original Score variable has the number “0” as the second character, in accordance with the 
achievement item naming convention (e.g., M051064A); 

• The Second Score variable has the letter “R” as the second character (e.g., MR5 1064 A); and 

• The Score Agreement variable has the letter “I” as the second character (e.g., MI5 1064A). 

Scoring Reliability Variable Score Values 

The values contained in both the Original Score and Second Score variables are the two-digit 
diagnostic codes assigned using the TIMSS scoring guides. The Score Agreement variable may have 
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one of three values, depending on the degree of agreement between the two scorers, as described in 
Exhibit 4.7. 


Exhibit 4.7: 

TIMSS 201 1 Score Agreement Variable Codes 

Code 

Description 

0 

Identical codes (both digits in the original and second scores) 

1 

Identical score levels, but different diagnostic codes (first digit of both 
scores are the same; second digits are different) 

2 

Different score levels (first digit of both scores are different) 


In general, the data in the Original Score variables are identical to those contained in the student 
achievement data files. In some cases, however, the response scores for specific items were recoded 
after a review of the international item statistics revealed inconsistencies in the original scoring 
guides, or showed that the original scores were not functioning as desired. The recoded score values 
were used in computing the achievement scores reflected in the International Reports and are present 
in the student achievement data files. In contrast, the Original Score variables in the scoring 
reliability data files contain the original unrecoded response scores. This was done so that the scoring 
reliability measures indicated in the Score Agreement variables were based on the original scoring 
guides used during the constructed response scoring sessions conducted in each country. 

4.4.3 TIMSS2011 Background Questionnaire Data Files 

There are five usual types of TIMSS 2011 background questionnaire data files: four data files 
(student, teacher, school, and curriculum) correspond to the four types of background questionnaires 
administered in TIMSS 2011, and the fifth links the student and teacher background data. The first 
four types of data files contain the responses to the questions asked in their respective background 
questionnaires. 

At the fourth grade, there is an additional file type that contains the responses from parents to the 
Learning to Read survey administered as part of the PIRLS 2011 assessment. This questionnaire was 
used in countries that administered the TIMSS and PIRLS 2011 assessments to the same sample of 
students; consequently, these countries have an additional TIMSS 2011 background questionnaire 
data file to hold these data. 

TIMSS 2011 Student Background Data Files (ASG/BSG) 

Students who participated in TIMSS 2011 were administered a background questionnaire with 
questions related to their home background, school experiences, and attitudes to mathematics and 
science. The student background data files contain students’ responses to these questions. They also 
contain students’ mathematics and science achievement scores (plausible values) to facilitate 
analyses of relationships between student background characteristics and achievement. 

Two versions of the student questionnaire were administered at the eighth grade. One version 
was for educational systems where science is taught as an integrated subject (integrated science 
version). The other version was for educational systems where the sciences (biology or life science, 
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physics, chemistry, and earth science) are taught separately (separate science version). For eighth 
grade students who were administered the integrated science version, questions that were given only 
in the separate science version were coded as “Not Administered.” For students who were assigned 
the separate science version, questions that were asked only in the integrated science version were 
coded as “Not Administered.” At the fourth grade, there was a single version of the student 
questionnaire, tailored toward general science. 

The student background data files also contain a number of identification variables, tracking 
variables, sampling and weighting variables, and derived variables that were used for producing 
exhibits in the international reports. These variables are described later in this chapter. 

PIRLS 2011 Home Background Data Files (ASH) 

Countries that administered the TIMSS and PIRLS 2011 fourth grade assessments to the same 
sample of students also administered a “Learning to Read Survey,” which was completed by the 
students’ parents or guardians. This survey usually is called the Home Questionnaire. It asked 
questions about preparations for primary schooling, including attendance in preschool and literacy- 
centered activities in the home before the child began school, such as reading books, singing songs, 
or writing letters or words. Parents answered questions about home resources in addition to 
infonnation about their highest level of education and employment situations. 

TIMSS 20 7 7 Teacher Background Data Files (A TG/BTM/BTS) 

The mathematics and science teachers of the students that were sampled for TIMSS 2011 were 
administered at least one questionnaire with questions pertaining to their background and their 
teaching practices in the classes of the sampled students. Each teacher was asked to respond to a 
questionnaire for each class taught that contained sampled students. The teacher background data 
files contain one record for each of the classes taught either by a mathematics or a science teacher. 

There were two types of teacher questionnaires administered at the eighth grade: one for the 
mathematics teachers, and one for the science teachers. The responses of teachers to the mathematics 
questionnaire are found in the BTM files and the responses of teachers to the science questionnaire 
are found in the BTS files. Variable names for questions asked in both questionnaires are the same. 
At the fourth grade, the situation was more straightforward, with a single teacher questionnaire 
requesting information on both mathematics and science, and all teachers’ responses are found in the 
ATG files. 

In the teacher background data files at both grades, each teacher has a unique identification 
number (IDTEACH) and a link number (IDLINK) that is specific to the class taught by the teacher 
and to which the information in the data record corresponds. The IDTEACH and IDLINK 
combination uniquely identifies, within a country, a teacher teaching a specific class. Thus, students 
linked to teachers identified by the same IDTEACH but different IDLINK are taught by the same 
teacher but in different classes. The teacher background data files cannot be merged directly with the 
student data files, and they do not contain sampling and weighting information, nor achievement 
scores. 

It is important to note that the teachers in the teacher background data files do not constitute a 
representative sample of teachers in a country, but rather are the teachers who taught a representative 

THE TIMSS 201 1 INTERNATIONAL DATABASE FILES 

CHAPTER 4 89 


TIMSS & PIRLS 

International Study Center 

Lynch School of Education, Boston College 



sample of students. The teacher data, therefore, should be thought of as attributes of the students to 
which they are linked, and should be analyzed only in conjunction with the student-teacher linkage 
data files. Chapters 2 and 3 of this User Guide describe student-level analyses combining the teacher 
data and the student-teacher linkage data files with the IEA IDB Analyzer software, as well as with 
SAS programs. 

TIMSS 2011 School Background Data Files (ACG/BCG) 

The school background data files contain school principals’ responses to the questions in the 
TIMSS 2011 school background questionnaires. Although school-level analyses where the schools 
are the units of analysis can be performed, it is preferable to analyze school-level variables as 
attributes of students. To perform student-level analyses with school data, the school background 
data files must be merged with the student background data files using the country and school 
identification variables. Details of the merging procedure with the IEA IDB Analyzer, or SAS 
programs, are described in Chapters 2 and 3 of this User Guide, respectively. 

TIMSS 2011 Student-Teacher Linkage Data Files (AST/BST) 

The TIMSS 2011 student-teacher linkage data files contain information required to link the student 
and teacher data files. The student-teacher linkage data files contain one entry per student-teacher 
linkage combination in the data. For instance, if three teachers are linked to a student, there are three 
entries in the file corresponding to that student. The sole purpose of the student-teacher linkage data 
files is to link teacher-level data with student-level data to perform appropriate student-level analyses 
where teacher characteristics are considered as attributes of the students. 

TIMSS 2011 Curriculum Data Files 

The TIMSS 2011 curriculum questionnaire data files contain the responses provided by the National 
Research Coordinators of the participating countries to the TIMSS 2011 curriculum questionnaires. 
There are two separate curriculum questionnaire data files for the two grades, fourth and eighth. 
These files are available as Excel files in the TIMSS 2011 International Database. 

Background Variable Naming Convention 

The background variable naming convention is based on a 7- or 8-character string. The following 
rules are applied in naming the background variables: 

• The first character is either “A,” for fourth grade data, or “B,” for eighth grade data. 

• The second character indicates the type of respondent. The letter “C” is used to identify data from 
the school principals, the letter “T” is used for teacher data, the letter “S” for student and parent 
data. 

• The third character is used to indicate the source of the data. The letter “B” is used for all 
background variables associated with questions in the background questionnaires and for the 
Rasch scores to contextual scales constructed from these data. The letter “D” is used for all 
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variables derived from responses in the background questionnairesand for the index variables 
constructed from these background Rasch scores. 6 

• The fourth character is used to indicate the subject or topic to which a background question 
refers. The following letters are used: 7 8 

G — General questions (not subject specific); 

H — Home questionnaire questions; 

M — Questions related to mathematics; 

S — Questions related to science; 

B — Questions related to biology or life science; 

C — Questions related to chemistry; 

E — Questions related to earth science; and 
P — Questions related to physics of physical science. 

• The fifth through eighth characters of all background questionnaire variables represent the 

o 

sequential numbering of the questions as presented in their respective questionnaires. 

The curriculum questionnaires use a different variable naming convention. The first three 
characters are “GEN” for general questions, “MA4” and “MA8” for fourth grade and eighth grade 
mathematics questions, respectively, and “SC4” and SC8” for fourth grade and eighth grade sciecne 
questions, respectively. The remaining characters indicate the sequential order of the questions in the 
curriculum questionnaires. 

Background Variable Location Convention 

Although the background variable naming convention was modified in 201 1 to show explicitly the 
ordering of questions in the background questionnaires, each question also was assigned a unique 
location code as shown in Exhibit 4.8. This unique code includes the sequence number of the 
question within the questionnaire, the same sequence number now found in the question’s variable 
name. For example, if the location variable is given as SQG-06, it refers to general question 6 in the 
student background questionnaire. This convention is followed in the data almanacs and in the 
description of the variables included in Supplements 1 and 2 to this User Guide. 


6 The contextual scales are described in the context questionnaire scales section of Methods and Procedures in TIMSS and PIRLS 2011. 

7 The letters “B,” “C,” “E,” and “P” are used only in the eighth grade student background data files for variables corresponding to questions about 
separate sciences asked in the separate science version of the student questionnaire. 

8 Because of the modular structure of the background questionnaires, the sequential numbering can be specific to each module. 
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Exhibit 4.8: Background Questionnaire Variable Location Convention 


Questionnaire 

Location Variable 

Student Questionnaire 

SQG-««« 

SQMS-»»» 

SQIS-»»» 

SQSS-»»» 

Fourth grade general questions 
Fourth grade mathematics and science questions 
Eighth grade integrated science questionnaire 
Eighth grade separate science questionnaire 

Home Questionnaire 

HQ-»»» 

Fourth grade 

Teacher Questionnaire 

TQG-»»» 

TQM-«»» 

TQS-««« 

Both grades general questions 
Both grades mathematics questions 
Both grades science questions 

School Questionnaire 

SCQ-»»» 

Both grades 

Curriculum Questionnaire 

CQG-»»» 

CQM4-*** 

CQS4-««« 

CQM8-»»» 

CQS8-««« 

Both grades general questions 
Fourth grade mathematics questions 
Fourth grade science questions 
Eighth grade mathematics questions 
Eighth grade science questions 


••• = sequential numbering of the question location in the questionnaire 


Data Coding Conventions 

A series of conventions also were adopted to code the data included in the background data fdes. 
This section describes these conventions. 

Background Question Response Code Values 

The values assigned to each of the background variables depend on the item format and the number 
of options available. For categorical questions, sequential numerical values were used to correspond 
to the response options available. The numbers correspond to the sequence of appearance of the 
response options. For example, the first response option was represented with a 1, the second 
response option with a 2, etcetera. Open-ended questions, such as “the number of students in a 
class,” were coded with the actual number given as a response. 

Codes for Missing Values 

A subset of values is reserved for specific codes related to various categories of missing data. 
Omitted Response Codes (SAS: . ; SPSS: 9, 99, 999, ...) 

“Omitted” response codes were used for questions that a student, parent, teacher, or school principal 
should have answered but did not. The length of the omitted response code given to a variable in the 
SPSS data files depends on the number of characters needed to represent the variable. In all cases, 
the space necessary to represent the variable is filled with 9’s. For questionnaire data, no distinction 
was made between items left blank and items with invalid answers, such as checking two or more 
response options in a categorical question, or unreadable or uninterpretable responses to open-ended 
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questions. In a small number of cases, data received from a country in an invalid or inconsistent 
manner also were coded as “Omitted.” 

Not Administered Response Codes (SAS: .A ; SPSS: sysmis) 

Special codes were given to items that were “Not Administered,” to distinguish these cases from data 
that were missing due to non-response. In general, the “Not Administered” code was used when an 
entire questionnaire was not completed or a specific question was not administered, such as when a 
question was left out of the instrument or misprinted. The “Not Administered” code was used in the 
following cases: 

• Background question left out or misprinted — When a particular question (or a whole page) was 
misprinted, or otherwise not available to the respondent, the corresponding variables were coded 
as “Not Administered.” 

• Background questions removed — Variables corresponding to questions in the student, home, 
teacher, or school background questionnaires that were considered not applicable in some 
countries were not included in the national versions of the questionnaires. These questions were 
coded as “Not Administered.” 

• Background questions mistranslated or not internationally comparable — In some cases, questions 
in the international version of the questionnaires were mistranslated or modified to fit the 
national context. Whenever possible, modified questions were recoded to match as closely as 
possible the international version. When this was not possible, modified questions were coded as 
“Not Administered.” 

Not Applicable Response Codes (SAS: .B ; SPSS: 6, 96, 996, ...) 

“Not Applicable” response codes were used for the background questionnaire items for which 
responses were dependent upon a filter question. Generally, a “No” response to a filter question lead 
to any follow-up questions being coded as “Not Applicable” because there were no appropriate 
responses to these follow-up questions. 

Contextual Scales and Derived Variables 

In the TIMSS 2011 questionnaires, often several questions were asked about various aspects of a 
single contextual construct. In these cases, responses to the individual items were combined to create 
a score, using Rasch scaling, which provided a more comprehensive picture of the contextual 
construct of interest than the individual variables could on their own. These Rasch scores also were 
categorized, usually into three groups, to create an index. The Rasch scores and their indices are 
included in the International Database and described in the contextual questionnaire scaling section 
of Methods and Procedures in TIMSS and PIRLS 2011. 

Additional variables were derived from responses to multiple questions to provide more pertinent 
infonnation for reporting. Parents’ education is an example where responses from both parents were 
combined into a single variable in order to report a single educational level. Generally, records 
(whether students, parents, teachers, or schools) were included in the calculation only if data were 
available for at least two-thirds of the variables involved. For example, if a derived variable was 
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based on six component variables, records that were missing responses to more than two of these 
were counted as missing on the derived variable. Supplement 3 to the User Guide provides a 
description of the derived variables included in the International Database. 

Sampling and Weighting Variables 

Several sampling and weighting variables are included in the TIMSS 2011 data files. They are listed 
and described in Exhibit 4.9 and Exhibit 4.10 illustrates the location of the various sampling and 
weighting variables among the different types of data files. It is important to note that the teacher 
background data files do not have any sampling and weighting variables, nor do the home 
background data files. 

Exhibit 4.9: TIMSS 201 1 Sampling and Weighting Variables 


Variable Name 

Description 

JKZONE 

The sampling zone, or stratum, to which the student's school is 
assigned 

JKREP 

The sampling replicate, or primary sampling unit, to which the 
student's school is assigned 

JKCZONE 

The sampling zone, or stratum, to which the school is assigned 

JKCREP 

The sampling replicate, or primary sampling unit, to which the 
school is assigned 

TOTWGT 

Total student weight— sums to the national population 

SENWGT 

Student senate weight— sums to 500 in each country 

HOUWGT 

Student house weight— sums to the national student sample size 

SCHWGT 

School weight 

TCHWGT 

Overall teacher weight 

MATWGT 

Mathematics teacher weight 

SCIWGT 

Science teacher weight 

WGTFAC1 

School weighting factor 

WGTADJ1 

School weighting adjustment 

WGTFAC2 

Class weighting factor 

WGTADJ2 

Class weighting adjustment 

WGTFAC3 

Student weighting factor 

WGTADJ3 

Student weighting adjustment 


Although TOTWGT has desirable properties, it also has drawbacks for some analyses. Because 
TOTWGT sums to the student population size in each country, analyses using TOTWGT that 
combine countries will have proportionately more students from larger countries and fewer from 
smaller countries, which may not be desirable for some purposes. For cross-country analyses in 
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which countries should be treated equally, TIMSS provides SENWGT, a transformation of 
TOTWGT, that results in a weighted sample size of 500 in each country. Additionally, because 
TOTWGT inflates sample sizes to estimate the population size, software systems that use the actual 
sample size to compute significance tests will give misleading results for analyses weighted by 
TOTWGT. HOUWGT, another transformation of TOTWGT, ensures that the weighted sample 
corresponds to the actual sample size in each country. 

Exhibit 4.1 0: Location of Sampling and Weighting Variables in the TIMSS 2011 Data Files 


Sampling and Data File Type 

Weighting ASG ASA AST ACG 

Variable BSG BSA BST BCG 


JKREP 


JKZONE 


JKCREP 


JKCZONE 


TOTWGT 


SENWGT 


HOUWGT 


SCHWGT 


TCHWGT 


MATWGT 


SCIWGT 


WGTFAC1 


WGTADJ1 


WGTFAC2 


WGTADJ2 


WGTFAC3 


WGTADJ3 


The weight variables TOTWGT, SENWGT, and HOUWGT are designed for use in student-level 
analyses from all student, home, and school background data files. The weight variable SCHWGT is 
designed for use in school-level analyses where the schools are the units of analysis. 

The weight variables TCHWGT, MATWGT, and SCIWGT are specifically designed for using 
teacher background data in student-level analyses and are based on TOTWGT. Whereas TCHWGT 
is used for analyses using all teachers, MATWGT and SCIWGT are used for analyses of 
mathematics and science teachers, respectively. These teacher weights are located in the student- 
teacher linkage files (AST and BST), not in the actual teacher background data files (ATG, BTM, 
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and BTS). Analyses with teacher data will be properly weighted by merging the teacher files with the 
student-teacher linkage files. 

The sampling variables beginning with the letters “JK” are used to compute standard errors based 
on the jackknife repeated replication methodology. All weighting variables beginning with the letters 
“WGT” are included to provide insight into the multi-stage sampling and weighting methodology 
applied to the TIMSS data. All weighting variables are described in the sampling section of Methods 
and Procedures in TIMSS and PIRLS 201 1. 

Structure and Design Variables 

Besides the variables used to store responses to the background questionnaires and achievement 
booklets, the TIMSS 2011 data files also contain variables meant to store information that identify 
and describe the respondents and design information required to properly analyze the data. 

Identification Variables 

In all TIMSS data files, several identification variables are included that provide information to 
identify countries, students, teachers, or schools. These variables also are used to link cases between 
the different data file types. The identification variables have the prefix “ID” and are described 
below. 

IDCNTRY 

IDCNTRY is a five-digit country identification code based on the ISO 3166 classification as shown 
in Exhibit 4.1. This variable should always be used as the first linking variable whenever files are 
li nk ed within and across countries. 

IDPOP 

IDPOP identifies the target grade and is set to “1” for the fourth grade and “2” for the eighth grade. 
IDGRADE 

IDGRADE identifies the target grade of the participating students. In TIMSS 2011, the usual values 
are “4” and “8” for most countries. 

IDSCHOOL 

IDSCHOOL is a four-digit identification code that uniquely identifies the participating schools 
within each country. The school codes are not unique across countries. Schools across countries can 
be uniquely identified only with the IDCNTRY and IDSCHOOL combination of linking variables. 

IDCLASS 

IDCLASS is a six-digit identification code that uniquely identifies the sampled classrooms within a 
country. The variable IDCLASS has a hierarchical structure and is formed by concatenating the 
IDSCHOOL variable and a two-digit sequential number identifying the sampled classrooms within a 


TIMSS 201 1 USER GUIDE FOR THE INTERNATIONAL DATABASE 

96 CHAPTER 4 



TIMSS & PIRLS 

International Study Center 

Lynch School of Education, Boston College 



school. Classrooms can be uniquely identified in the database by the combination of IDCNTRY and 
IDCLASS as linking variables. 

IDSTUD 

IDSTUD is an eight-digit identification code that uniquely identifies each sampled student in a 
country. The variable IDSTUD also has a hierarchical structure and is formed by concatenating the 
IDCLASS variable and a two-digit sequential number identifying all students within each classroom. 
Students can be uniquely identified in the database by the combination of IDCNTRY and IDSTUD 
as linking variables. 

IDBOOK 

IDBOOK identifies the specific assessment booklet that was administered to each student. 

The booklets are given a numerical value from “1” through “14.” 

IDSTRATE & IDSTRATI 

IDSTRATE and IDSTRATI are identification variables generated by the school sampling process. 
IDSTRATE identifies the explicit strata and IDSTRATI the implicit strata from which the 
participating schools were sampled. The codes assigned to these two variables vary from country to 
country and are documented in Supplement 4 to the User Guide. 

IDTEACH 

IDTEACH is a six-digit identification code that uniquely identifies a teacher within a school. It has a 
hierarchical structure and is formed by the concatenation of IDSCHOOL and a two-digit sequential 
number within each school. 

IDLINK 

IDLINK uniquely identifies the class for which a teacher answered a questionnaire. The combination 
of linking variables IDCNTRY, IDTEACH, and IDLINK uniquely identifies all teacher-class 
combinations in the database. 

Exhibit 4.11 shows in which data files the various identification variables are located. It also 
highlights the combinations of variables used to uniquely identify the records contained in the 
different data file types. In the student background, home background, and achievement data files, 
the variables IDCNTRY and IDSTUD provide a unique identification number to identify all students 
in the database. Because teachers may teach more than one class, the combination of the IDCNTRY, 
IDTEACH, and IDLINK variables in the teacher background data files is needed to uniquely identify 
all teachers and the classes they teach. Teacher background variables are linked to the appropriate 
students using the student-teacher linkage data files. The variable IDSCHOOL, contained in all files, 
is a unique identification number for each school within a country. Combined with IDCNTRY, it can 
be used to link school background data to corresponding students or teachers. 
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Exhibit 4.1 1 : Location of Identification Variables in the TIMSS 2011 Data Files 


Data File Type 


lUCIlUIILdllUll 

Variable 

ASA 

BSA 

ASG 

BSG 

AST 

BST 

ATG 

BTM/BTS 

ACG 

BCG 

ASH 

IDCNTRY 

• 

• 

• 

• 

• 

• 

IDGRADE 

• 

• 

• 

• 

• 

• 

IDPOP 

• 

• 

• 

• 

• 

• 

IDSCHOOL • • • • 

• 

• 

IDCLASS • • • • • 

IDSTUD • • • 

• 

IDBOOK • • 

IDSTRATE • • 

IDSTRATI • • 

IDTEACH 

• 

• 


IDLINK 

• 

• 



Tracking Variables 

Information about students, teachers, and schools provided by the survey tracking forms is stored in 
the tracking variables. 9 These variables have the prefix “IT.” All tracking variables are included in 
the student background data files. ITLANG is included in the student achievement data files, student 
background data files, and home background data files. 

ITSEX 

Gender of each student as stated in the Student Tracking Forms. 

ITBIRTHM and ITBIRTHY 

Month and year of birth of each student as stated in the Student Tracking Forms. 

ITDATEM and ITDATEY 

Month and year of testing for each student. 

ITLANG 

Language of testing for each student. It is set to “1” for all countries that tested in a single language. 
For countries that administered the test in more than one language, additional numerical codes are 
used that correspond to the order of the testing languages as shown in Supplement 2 to the User 
Guide. 


9 


Survey tracking forms are lists of students, teachers, and schools used for sampling and administrative purposes. 
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4.5 TIMSS 2011 Codebook Files 


All information related to the structure of the TIMSS 2011 data files, as well as the source, format, 
descriptive labels, and response option codes for all variables, is contained in codebook files. Each 
data file type in the International Database is accompanied by a codebook file, with the exception of 
the curriculum data files. The naming convention for codebook files is as follows: 

• The first three characters of the filename are in every respect identical to those in the file names 
shown in Exhibit 4.2; 

• The next three characters identify the files as TIMSS codebooks and are always “TMS”; 

• The seventh and eighth characters are always “M5” to indicate the TIMSS 2011 study cycle; and 

• The three-character file extension is either .XLS for the Excel format, or .PDF for the printable 
format. 

Codebook files can be read using Excel, or printed directly from PDF format using Acrobat 
Reader. They describe the contents and structure of the TIMSS data files. Important codebook fields 
include FIELD LABL, which contains extended textual information for all variables, QUEST LOC, 
which provides the location of questions and achievement items within their respective survey 
instruments, and FIELD CODE, which lists all acceptable responses allowed in the database. 

4.6 TIMSS 2011 Data Almanac Files 

Data almanacs provide weighted summary statistics for all variables in the TIMSS 201 1 data files. 
There are two basic types of data almanacs: achievement data almanacs for the achievement items, 
and background data almanacs for the background variables. All data almanac files are provided in 
Word format and printable PDF format. 

Achievement Data Almanacs 

The achievement data almanacs provide weighted summary statistics for each participating country 
on each individual achievement item included in the TIMSS 2011 assessment. There are separate 
achievement data almanacs for fourth and eighth grades and for the two subjects, mathematics and 
science. The achievement data almanacs display for each item its classification in the content and 
cognitive domains, the item block to which it belongs, a brief description of the item, its variable 
name, whether it is a multiple choice or constructed response item, the correct response key if it is a 
multiple choice item, and its point value if it is a constructed response item. The trend item almanacs 
provide summary statistics for achievement items used in both the 2007 and 2011 assessments. The 
achievement data almanac files available in the International Database are listed in Exhibit 4.12. 

The achievement data almanacs also display the international averages for each item, with each 
country weighted equally. The out-of-grade countries and benchmark participants, listed below the 
international averages, are not included in the calculation of international averages. 
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Exhibit 4.1 2: TIMSS 2011 Achievement Data Almanacs 


Achievement Data Almanac Contents 


Tll_G4_MAT_ItemAlmanac Almanac for fourth grade mathematics items 

Tll_G4_SCI_ItemAlmanac Almanac for fourth grade science items 

Tll_G4_MAT_TrendItemAlmanac Trend almanac for fourth grade mathematics items 
Tll__G4_SCI_TrendItemAlmanac Trend almanac for fourth grade science items 


Tll_G8_MAT_ItemAlmanac Almanac for eighth grade mathematics items 

Tll__G8_SCI_ItemAlmanac Almanac for eighth grade science items 

Tll_G8_MAT_TrendItemAlmanac Trend almanac for eighth grade mathematics items 
Tll_G8_SCI_TrendItemAlmanac Trend almanac for eighth grade science items 


There are two types of displays in the achievement data almanacs, depending on whether an item 
is a multiple choice item or a constructed response item. The statistics displayed in these almanacs 
are as follows: 

• N — The number of students to whom the item was administered. 

• DIFF — Percent of students that responded correctly to a multiple choice item. 

• A, B, C, and D — The percent of students choosing each one of the response options for a 
multiple choice item. 

• Scoring Guide Codes (e.g., 10, 1 1, 70, 71) — The percent of student responses assigned each of 
the codes in the scoring guide for a constructed response item. 

• OMITTED — The percent of students that omitted, or did not to respond to, the item. 

• NOT REACHED — The percent of students that did not reach the item. 

• VI and V2 — The percent of students that scored 1 point or better on the item (V 1) or 2 points 
(V2). 

• 1 .GIRL %RIGHT and 2. BOY %RIGHT — The percent of girls and boys that either got a 
multiple choice item correct, or obtained the maximum score on a constructed response item. 

Background Data Almanacs 

Background data almanac files contain weighted summary statistics for each participating country on 
each variable in the student, home, teacher, and school background questionnaires, including the 
contextual scales and their indices and the derived variables based on these background variables. 10 
Among the statistics reported are mean mathematics and science achievement by response category. 
The background data almanacs also display for each variable the question as it was asked, its 
location in the corresponding questionnaire, and its variable name in the data files. The background 
data almanac files available in the TIMSS 2011 International Database are listed in Exhibit 4.13. 


10 


The Home Questionnaire almanac lists only the countries that administered the TIMSS and PIRLS 2011 fourth grade assessments to the same 
sample of students. 
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Exhibit 4.1 3: TIMSS 2011 Background Data Almanacs 


Background Data Almanac 

Contents 

Tll_G4_MAT_StudentAlmanac 

Fourth grade student background almanac 
with mathematics achievement 

Tll_G4_SCI_StudentAlmanac 

Fourth grade student background almanac 
with science achievement 

Tll_G4_MAT_HomeAlmanac 

Fourth grade home background almanac with 
mathematics achievement 

Tll_G4_SCI_HomeAlmanac 

Fourth grade home background almanac with 
science achievement 

Tll_G4_MAT_TeacherAlmanac 

Fourth grade teacher background almanac 
with mathematics achievement 

Tll_G4_SCI_TeacherAlmanac 

Fourth grade teacher background almanac 
with science achievement 

Tll_G4_MAT_SchoolAlmanac 

Fourth grade school background almanac with 
mathematics achievement 

Tll_G4_SCI_SchoolAlmanac 

Fourth grade school background almanac with 
science achievement 

Tll_G8_MAT_StudentAlmanac 

Eighth grade student background almanac 
with mathematics achievement 

Tll_G8_SCI_StudentAlmanac 

Eighth grade student background almanac 
with science achievement 

Tll_G8_MAT_TeacherAlmanac 

Eighth grade teacher background almanac 
with mathematics achievement 

Tll_G8_SCI_TeacherAlmanac 

Eighth grade teacher background almanac 
with science achievement 

Tll_G8_MAT_SchoolAlmanac 

Eighth grade school background almanac with 
mathematics achievement 

Til G8 SCI SchoolAlmanac 

Eighth grade school background almanac with 


science achievement 


The background data almanacs also display the international averages for each variable, with 
each country weighted equally. The out-of-grade countries and benchmark participants, listed below 
the international averages, are not included in the calculation of international averages. 

There are two types of displays in the background data almanacs, depending on whether the data 
are categorical (i.e., have a small number of discrete values) or continuous. The almanac display for 
categorical variables includes the following: 

• The sample size (number of students, parents, teachers, or schools included in the sample); 
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• The number of valid cases (number of students, parents, teachers, or schools for whom valid data 
were obtained); 

• The weighted percentages of students corresponding to each valid response option (percentages 
based only on the students with valid data, as well as “Not Applicable” codes when used); 

• The weighted percentages of students for whom none of the valid response options were selected, 
coded as “Not Administered” or “Omitted” (percentages based on the sample size); 

• The weighted mean achievement values of students corresponding to each valid response option, 
as well as the “Not Administered” and “Omitted” codes; and 

• In cases where a variable can be coded as “Not Applicable” because of an earlier filter question, 
the weighted percentage of students for whom the variable is coded as “Not Applicable” also is 
displayed, along with the corresponding weighted mean achievement. 

The almanac display for continuous variables includes the following: 

• The sample size (number of students, parents, teachers, or schools included in the sample); 

• The number of valid cases (number of students, parents, teachers, or schools for whom valid data 
were obtained); 

• The weighted percentages of students for whom the variable is coded as “Not Administered” or 
“Omitted” (percentages based on the sample size); 

• The weighted mean, mode, minimum, maximum, and the 5th, 10th, 25th, 50th, 75th, 90th, and 
95th percentiles across students (based only on the students with valid data, as well as “Not 
Applicable” codes when used); and 

• In cases where a variable can be coded as “Not Applicable” because of an earlier filter question, 
the weighted percentage of students for whom the variable is coded as “Not Applicable” also is 
displayed. 

It is important to note that all percentages and distributional statistics of continuous variables 
reported in the background data almanacs always are based on student-level calculations — for 
example, the percentage of students whose teachers or schools gave a particular response to a 
question, because teacher data and school data usually were analyzed as student attributes. 

4.7 TIMSS 201 1 Test-Curriculum Matching Analysis Data Files 

The Test-Curriculum Matching Analysis (TCMA) was conducted to investigate the appropriateness 
of the TIMSS 2011 mathematics and science assessments for the fourth and eighth grade students in 
the participating countries. To that end, participating countries were asked to indicate which items on 
the TIMSS 2011 assessments were included in their national curricula. Thus, based on computations 
of average percent correct, each country was able to see the performance of all countries on the items 
appropriate for its curriculum, and also the performance of its students on the items judged 
appropriate for the curriculum in other countries. The analytical method used and the results of the 
TCMA are presented in Appendix F of the TIMSS 2011 International Reports. 

The International Database contains four TCMA data files — one for each combination of grade 
and subject — in Excel format and printable PDF format, showing which items were selected by each 
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participating country. Only those countries that submitted TCMA item selection information are 
included in the fdes and were presented in the reported results. 

4.8 TIMSS 201 1 Program Files 

The TIMSS 2011 International Database includes a number of SAS programs and macros designed 
to facilitate the manipulation of the TIMSS 2011 data files and conduct proper statistical analyses 
taking into account the jackknife algorithm and the presence of plausible values. These programs are 
described in Chapter 3 of this User Guide. 

The International Database contains two SPSS syntax files — ASASCRM5.SPS and 
BSASRCM5.SPS — which will recode the responses to individual items from the TIMSS 2011 
achievement data files to their appropriate score levels. The files function much the same way as 
their SAS counterparts described in Chapter 3 of this User Guide (see Section 3.4). The International 
Database also includes a third SPSS syntax file to perform a variable recode required for the proper 
execution of the third example in Chapter 2 of this User Guide (see Section 2.5). 
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Appendix 

Organizations and Individuals 
Responsible for TIMSS 201 1 


TIMSS 2011 was a collaborative effort involving hundreds of individuals around 
the world. This appendix acknowledges the individuals and organizations for 
their contributions. Given that work on TIMSS 2011 has spanned approximately 
four years and has involved so many people and organizations, this list may 
not include all who contributed. Any omission is inadvertent. TIMSS 2011 
also acknowledges the students, parents, teachers, and school principals who 
contributed their time and effort to the study. This report would not be possible 
without them. 


Management and Coordination 

TIMSS is a major undertaking of IEA, and together with the Progress in 
International Reading Literacy Study (PIRLS) comprises the core of IEA’s regular 
cycles of studies. The TIMSS assessment at the fourth grade complements 
PIRLS, which regularly assesses reading achievement at the fourth grade. 

The TIMSS & PIRLS International Study Center at Boston College has 
responsibility for the overall direction and management of the TIMSS and PIRLS 
projects. Headed by Executive Directors Drs. Ina VS. Mullis and Michael O. Martin, 
the study center is located in the Lynch School of Education. In carrying out 
the project, the TIMSS & PIRLS International Study Center worked closely 
with the IEA Secretariat in Amsterdam, which managed country participation, 
was responsible for verification of all translations produced by the participating 
countries, and coordinated the school visits by International Quality Control 
Monitors. The IEA Data Processing and Research Center in Hamburg was 
responsible for processing and verifying the data submitted by the participants; 
Statistics Canada in Ottawa was responsible for school and student sampling 
activities; and Educational Testing Service in Princeton, New Jersey consulted 
on psychometric methodology, provided software for scaling the achievement 
data, and replicated the achievement scaling for quality assurance. 
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The Project Management Team, comprising the study directors and 
representatives from the TIMSS & PIRLS International Study Center, IEA 
Secretariat and IEA Data Processing and Research Center, Statistics Canada, 
and ETS met twice a year throughout the study to discuss the study’s progress, 
procedures, and schedule. In addition, the study directors met with members of 
IEA’s Technical Executive Group twice yearly to review technical issues. 

To work with the international team and coordinate within-country 
activities, each participating country designates an individual to be the TIMSS 
National Research Coordinator (NRC). The NRCs have the challenging task 
of implementing TIMSS in their countries in accordance with the TIMSS 
guidelines and procedures. In addition, the NRCs provide feedback and 
contributions throughout the development of the TIMSS assessment. The 
quality of the TIMSS assessment and data depends on the work of the NRCs 
and their colleagues in carrying out the complex sampling, data collection, and 
scoring tasks involved. Continuing the tradition of exemplary work established 
in previous cycles of TIMSS, the TIMSS 2011 NRCs performed their many tasks 
with dedication, competence, energy, and goodwill, and have been commended 
by the IEA Secretariat, the TIMSS & PIRLS International Study Center, the 
IEA Data Processing and Research Center, and Statistics Canada for their 
commitment to the project and the high quality of their work. 

Funding 

Funding for TIMSS 2011 was provided primarily by the participating countries. 
The National Center for Education Statistics of the US Department of Education 
was a major funding partner, providing funding under contract number 
ED08C00117. The content of this publication does not necessarily reflect the 
views or policies of the US Department of Education nor does mention of 
trade names, commercial products, or organizations imply endorsement by 
the US Government. 

The financial support from Boston College is gratefully acknowledged, as 
well as that from the UK’s National Foundation for Educational Research. 
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IEA Secretariat 

Seamus Hegarty, IEA Chair 

Hans Wagemaker, Executive Director 

Barbara Malak, Manager, Member Relations 

Juriaan Hartenberg, Financial Manager 

Paulina Korsnakova, Senior Professional Researcher 

Isabelle Braun-Gemin, Financial Manager Assistant 

David Ebbs, Management Assistant 

Alana Yu, Management Assistant 

TIMSS & PIRLS International Study Center at Boston College 

Ina V.S. Mullis, Executive Director 
Michael O. Martin, Executive Director 

Pierre Foy, Director of Sampling, Psychometrics, and Data Analysis 

Paul Connolly, Director, Graphic Design and Publications 

Alka Arora, Assistant Research Director, TIMSS Mathematics 

Gabrielle M. Stanco, Assistant Research Director, TIMSS Science 

Kathleen T. Drucker, Assistant Research Director, PIRLS Reading 

leva Johansone, Assistant Research Director, Operations and Quality Control 

Marcie Bligh, Manager of Office Administration 

Bradley Brossman, Psychometrician (through 2012) 

Courtney E. Castle, Graduate Assistant 

Victoria A.S. Centurino, Graduate Assistant 

Susan Farrell, Senior Data Graphics Specialist 

Joseph Galia, Senior Statistician/ Programmer 

Christine Hoage, Manager of Finance 

Lee R. Jones, Chief Science Consultant 

Chad A. Minnich, Writer/Editor 

Jennifer Moher Sepulveda, Data Graphics Specialist 

Mario A. Pita, Senior Data Graphics Specialist 

Jyothsna Pothana, Statistician/Programmer 

Betty Poulos, Administrative Coordinator 

Corinna Preuschoff, Senior Research Specialist (through 2011) 

Moira A. Ragan, Graduate Assistant 
Ruthanne Ryan, Data Graphics Specialist 
Steven A. Simpson, Data Graphics Specialist 
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IEA Data Processing and Research Center 

Dirk Hastedt, Co-Director 

Juliane Hencke, Project Co-Manager, TIMSS and PIRLS data processing 

Oliver Neuschmidt, Project Co-Manager, TIMSS and PIRLS data processing 

Yasin Afana, Deputy Project Manager, TIMSS and PIRLS data processing 

Milena Taneva, Deputy Project Manager, TIMSS and PIRLS data processing 

Alena Becker, Research Analyst 

Christine Busch, Research Analyst 

Ralph Carstens, Senior Research Analyst 

Mark Cockle, Research Analyst 

Tim Daniel, Research Analyst 

Limiao Duan, Research Analyst 

Eugenio Gonzales, Unit Head, Research and Analysis 

Pamela Inostroza, Research Analyst 

Michael Jung, Research Analyst 

Maike Junod, Programmer 

Alexander Konn, Programmer 

Marta Kostek-Drosihn, Unit Coordinator 

Sabine Meinck, Research Analyst, Sampling 

Sebastian Meyer, Research Analyst 

Dirk Oehler, Research Analyst 

Moritz Otto, Programmer 

Devi Potham Rajendra Prasath, Programmer 

Daniel Radtke, Research Analyst 

Anke Sielemann, Research Analyst 

Harpreet Singh Choudry, Unit Head, Software 

Sabine Tieck, Research Analyst 

Caroline Vandenplas, Research Analyst 

Sabine Weber, Research Analyst 

Bettina Wietzorek, Meeting and Seminar Coordinator 

Meng Xue, Programmer 

Olaf Zuehlke, Research Analyst, Sampling 

Statistics Canada 

Marc Joncas, Senior Methodologist 
Sylvie LaRoche, Senior Methodologist 
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Educational Testing Service 

Matthias Von Davier, Research Director 

Edward Kulick, Research Director 

Meng Wu, Associate Psychometrician 

Jonathan Weeks, Associate Research Scientist 

Scott Davis, Data Analysis and Computational Research Specialist 

Yuxin (Christina) Tang, Principal Research Data Analyst 

Zhumei Guo, Senior Research Data Analyst 

Sampling Referee 

Keith Rust, Vice President and Associate Director of the Statistical Group, Westat, Inc. 
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Mathematics 


Kiril Bankov 
University of Sofia 

Bulgaria 

Karen Manriquez 
Ministry of Education 

Chile 

Fou-Lai Lin 

National Taiwan Normal University 

Chinese Taipei 

Khattab M. A. Abulibdeh 

National Center for Human Resources 
Development 

Jordan 

Christoph Selter 
Mathematics Department 
TU Dortmund University 

Germany 


Robert Garden 
New Zealand 

Liv Sissel Gronmo 

Department of Teacher Education and 
School Research 

ILS, University of Oslo 

Norway 

Mary Lindquist 

United States 

Hung-Hsi Wu 

University of California, Berkeley 

United States 
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Science 


Gabriela Noveanu 
Institute for Educational Sciences 
Curriculum Department 

Romania 

Galina Kovaleva 

Institute of Content and Methods of Education 

Center for Evaluating the Quality of 
General Education 

Russian Academy of Education 

Russian Federation 

Maria Pilar Jimenez Aleixandre 
Universidade de Santiago de Compostela 

Spain 

Wolfgang Dietrich 
National Agency for Education 

Sweden 

Gerald T. Wheeler 

National Science Teachers' Association 

United States 

TIMSS 201 1 ITEM DEVELOPMENT TASK FORCES 

Mathematics 

Alka Arora, Assistant Research Director, TIMSS Mathematics 
Graham Ruddock, TIMSS Mathematics Coordinator (NFER) 

Ina V.S. Mullis, TIMSS & PIRLS International Study Center Executive Director 
Mary Lindquist, SMIRC Representative 
Robert Garden, SMIRC Representative 

Berinderjeet Kaur, Mathematics Consultant, National Institute of Education, Singapore 
Science 

Gabrielle M. Stanco, Assistant Research Director, TIMSS Science 
Christine O’Sullivan, TIMSS Science Coordinator (K-12 Consulting) 

Helen Lye, TIMSS Science Consultant (ACER) 

Gerald T. Wheeler, SMIRC Representative 


Martina Kekule 

Charles University in Prague 

Czech Republic 

Jouni Viiri 

University of Jyvaskyla 

Finland 

Saule Vingeliene 

Educational Development Centre 

Lithuania 

Berenice Michels 

National Institute for Curriculum Development 

The Netherlands 

Mariam Mohammad Ahmed 

Evaluation Institute, Supreme Education Council 

Qatar 
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Questionnaire Item Review Committee 


Sue Thomson 

Australian Council for Educational Research 

Australia 

Josef Basl 

Czech School Inspectorate 

Czech Republic 

Naima Hassan 

National Center of Examinations and 
Educational Evaluation 

Egypt 

Linda Sturman 

National Foundation for Educational Research 

England 

Wilfried Bos 

Institute for School Development Research (IFS) 
TU Dortmund University 

Germany 

Clara Rosaline Anumel 
Inspectorate Division 
Ghana Education Service 

Ghana 


Frederick Leung 

Faculty of Education 

The University of Hong Kong 

Hong Kong SAR 

Martina Meelissen 

Department of Educational Organization and 
Management 

Faculty of Behavioral Sciences 
University of Twente 

Netherlands 

Barbara Japelj Pavesic 
Educational Research Institute 

Slovenia 

Peter Nystrdm 
Umea University 

Sweden 

Patrick Gonzales 

National Center for Education Statistics 
U.S. Department of Education 

United States 
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Armenia 

Arsen Baghdasaryan 

Assessment and Testing Center of Armenia 

Australia 

Sue Thomson 

Australian Council for Educational Research 

Austria 

Birgit Suchari 

Bundesinstitut fuer Bildungsforschung, 
Innovation und Entwicklung des 
Oesterreichischen Schulwesens (BIFIE) 

Azerbaijan 

Emin Meherremov 

Ulviya Mikailova (through 2010) 

Department of Monitoring and Assessment 
Ministry of Education 

Bahrain 

Huda Al-Awadi 
Ministry of Education 

Belgium (Flemish) 

Jan Van Damme 
Barbara Belfi 

Centrum voor Onderwijseffectiviteit en evaluatie 
Katholieke Universiteit Leuven 

Botswana 

Monamodi Kesamang 
Botswana Examinations Council 

Chile 

Johanna Gubler Santander 
Ministerio de Educacion 


Chinese Taipei 

Chen-Yung Lin 

Graduate Institute of Science Education 
National Taiwan Normal University 

Croatia 

Jasminka Buljan Culej 

National Center for External Evaluation in 
Education 

Czech Republic 

Vladislav Tomasek 
Czech School Inspectorate 

Denmark 

Peter Allerup 
Department of Education 
Aarhus University 

England 

Linda Sturman 

National Foundation for Educational Research 

Finland 

Pekka Kupari 

Finnish Institute for Educational Research 
University of Jyvaskyla 

Georgia 

Mamuka Jibladze 

Dito Pataraia (through 201 1 ) 

National Assessment and Examinations Center 

Germany 

Wilfried Bos 
Heike Wendt 

Institute for School Development Research (IFS) 
TU Dortmund University 
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Ghana 

Clara Rosaline Anumel 
Inspectorate Division 
Ghana Education Service 

Honduras 

Renan Rapalo Castellanos 
Secretaria de Educacion 
Instituto de Investigacion 
Universidad Pedagogica Nacional 

Hong Kong SAR 

Frederick Leung 

Faculty of Education 

The University of Hong Kong 

Hungary 

Ildiko Szepesi 
Educational Authority 

Department of Educational Assessment and 
Evaluation 

Indonesia 

Hari Setiadi 

Nugaan Yulia Wardani (through 2010) 
Center for Educational Assessment 
Ministry of National Education 

Iran, Islamic Republic of 

Abdol’azim Karimi 

Research Institute for Education (RIE) 
Ministry of Education 

Ireland 

Eemer Eivers 

Educational Research Centre 
St. Patricks College, Dublin 


Israel 

Inbal Ron-Kaplan 

National Authority for Measurement and 
Evaluation in Education (RAMA) 

Italy 

Elisa Caponera 

Instituto Nazionale per la Valutazione 
del Sistema Educativo di Instruzione e 
di Formazione (INVALSI) 

Japan 

Fumi Ginshima 

Kenji Matsubara 

Yasushi Ogura (through 2011) 

Keiichi Nishimura (through 201 1 ) 

Department for Curriculum Research 
Curriculum Research Center 

National Institute for Educational 
Policy Research (NIER) 

Jordan 

Khattab M. A. Abulibdeh 

National Center for Human Resources 
Development 

Kazakhstan 

Zhanat Bazarbekova 

Tynyshkul Moldashevna Amreeva (through 201 1 ) 
National Centre for Education Quality Assessment 

Korea, Republic of 

Soojin Kim 

Kyunghee Kim (through 2010) 

Korea Institute for Curriculum & Evaluation 

Kuwait 

Marzouq Al-Ghonaim 
Ministry of Education 
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Lebanon 

Leila Maliha Fayad 

Educational Center for Research and Development 
Ministry of Education 

Lithuania 

Olga Kostina 

Aiste Elijio (through 2011) 

National Examination Centre 
Ministry of Education and Science 

Macedonia 

Beti Lameva 

National Examination Center 

Malaysia 

Faridah Abu Hassan 
Dewani Goloi 

Muhammad Zaini Mohd Zain 
Educational Planning & Research Division 
Ministry of Education 

Malta 

Raymond Camilleri 

Directorate for Quality & Standards in Education 
Ministry of Education 

Morocco 

Mohammed Sassi 

Departement de l’Education Nationale 

Centre Nationale de l’Evaluation et 
des Examens 

Netherlands 

Mar j olein Drent 
Martina Meelissen 

Department of Educational Organization and 
Management 

Faculty of Behavioral Sciences 
University of Twente 


New Zealand 

Robyn Caygill 

Comparative Education Research Unit 
Ministry of Education 

Northern Ireland 

Patricia Wyers 

Statistics and Research Branch 
Department of Education 

Norway 

Liv Sissel Gronmo 

Department of Teacher Education and 
School Research 

ILS, University of Oslo 

Oman 

Zuwaina Saleh Al-Maskari 
Ministry of Education 

Palestinian National Authority 

Mohammed O. Matar Mustafa 
Assessment and Evaluation Deparment 
Ministry of Education and Higher Education 

Poland 

Krzysztof Konarzewski 
Polish Academy of Sciences 

Dominik Mytkowski (through 2010) 

Centralna Komisja Egzaminacyjna 

Portugal 

Ana Ferreira 

Education Statistics and Planning Office 
Ministry of Education 

Qatar 

Abdulsattar Mohammed Nagi 

Office of Student Assessment 

Evaluation Institute, Supreme Education Council 
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Romania 

Gabriela Noveanu 

Institute for Educational Sciences 

Curriculum Department 

Russian Federation 

Galina Kovaleva 

Institute of Content and Methods of Education 

Center for Evaluating the Quality of 
General Education 

Russian Academy of Education 

Saudi Arabia 

Saleh Alshaya 

International Studies & Testing Center (ISTC) 
Ministry of Education 

Serbia 

Slobodanka Gasic-Pavisica 
Institute for Educational Research 

Singapore 

Pik Yen Lim 

Chew Leng Poon 

Elaine Chua Ka-Yi (through 201 1 ) 

Research and Evaluation Section/ 

Planning Division 

Ministry of Education 

Slovak Republic 

Andrea Galadova 

National Institute for Certified Educational 
Measurements 

Slovenia 

Barbara Japelj Pavesic 
Educational Research Institute 

South Africa 

Vijay Reddy 

Human Sciences Research Council (HSRC) 


Spain 

David Cervera Olivares 

Jesus Dominguez Castillo (through 2012) 

National Institute of Educational Evaluation 
Ministry of Education, Culture and Sports 

Sweden 

Eva Lundgren 

Swedish National Agency for Education 

Syrian Arab Republic 

Omar Abou Awn 
Ministry of Education 

Thailand 

Precharn Dechsri 

The Institute for the Promotion of Teaching 
Science and Technology 

Tunisia 

Kameleddine Gaha 

National Centre for Pedagogical Innovation and 
Research in Education 

Turkey 

Nurcan Ate$ok Deveci 

General Directorate of Innovation and 
Educational Technologies 

Murat Yalcin (through 201 1 ) 

Halil Rahman Acar (through 2010) 

Educational Research & Development Directorate 
Ministry of National Education 

Ukraine 

Larisa Dvoretska 

Ukranian Center of Evaluation of 
Educational Quality 

Nataliia Prokopenko (through 201 1 ) 

Ministry of Education and Science of Ukraine 
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United Arab Emirates 

Nada Abu Baker Husain Ruban 
Assessment Department 
Ministry of Education 

United States 

Stephen Provasnik 

Patrick Gonzalez (through 201 1 ) 

National Center for Education Statistics 
U.S. Department of Education 

Yemen 

Tawfiq Ahmad Al-Mekhlafy 

Educational Research and Development Center 

Ministry of Education 


BENCHMARKING PARTICIPANTS 

Alberta, Canada 

Ping Yang 

Learner Assessment Branch 
Alberta Education 

Ontario, Canada 

Michael Kozlow 

Education Quality and Accountability Office 

Quebec, Canada 

Robert Marcotte 

Direction de la sanction des etudes 

Abu Dhabi, United Arab Emirates 

Shaikha Ali A1 Zaabi 

Abu Dhabi Education Council 

Dubai, United Arab Emirates 

Mariam A1 Ali 

Knowledge and Human Development Authority 

Alabama, United States 
California, United States 
Colorado, United States 
Connecticut, United States 
Florida, United States 
Indiana, United States 
Massachusetts, United States 
Minnesota, United States 
North Carolina, United States 
Patrick Gonzalez 

National Center for Education Statistics 
U.S. Department of Education 
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